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Abstract. In this paper, we find an explicit combinatorial criterion 
for the existence of a nonzero GL„_i (AT)-high weight vector of weight 
(Ai, . . ., Ai_i, \i— d, Xi+i, . . ■ , A n _i), where d < char(A") and K is an 
algebraically closed field, in the irreducible rational GL n (A')-module 
L n (Ai, . . ., A n ) with highest weight (Ai, . . . , A n ). For this purpose, new 
modular lowering operators are introduced. 



1. Introduction 

Let K be an algebraically closed field of characteristic p > 0. We denote 
by GL n {K) the general linear group of degree re over K. Let D n (K) and 
T n {K) denote the subgroups of GL n {K) consisting of all diagonal matrices 
and all upper triangular matrices respectively. We call the elements of Z n 
weights. Any weight (Ai,...,A n ) will be identified with the character of 
Bn(K) that takes diag(ii, . . . ,t n ) to t^ 1 ■■■t^ n . We shall understand the 
weight spaces of rational GL n (X)-modules always with respect to the torus 
Y) n {K). A vector v of a rational GL n (K)-module is called a GL n (K)-high 
weight vector if the line K ■ v is fixed by T n (K). We denote by X + (n) 
the subset of Z" consisting of all weakly decreasing sequences and call the 
elements of X + (n) dominant weights. 

For n > 1, the group GL n _i(K) will be identified with the subgroup of 
GL n (K) consisting of matrices having in the last row and the last column 
except the position of their intersection, where they have 1. In what follows, 
L n (A) denotes the irreducible rational GL„(/C)-module with highest weight 
A G X + (n) and denotes a nonzero vector of L n (X) of weight A, which we 
fix. Let [s..t], [s..t), (s..t], (s..t) denote the sets {x £ Z | s ^ x ^ t},{x £ 
Z | s ^ x < t},{x € Z | s < x ^ t},{x € Z | s < x < t} respectively. 

To formulate the main result of this paper, we introduce the strict partial 
order < on Z 2 and the subsets 2)^(i, n) and <t x (i, n) of Z 2 and Z respectively 
as follows: (a, b) < (x, y) holds if and only if a < x and b < y; for any A € Z™, 
we put 

2^(i,n) := {(*, h) G (i..n] X [l..d] \ t - i + Aj - A t - h = (mod p)}, 

£ A (i, re) := {s G (i..n) \ s — i + Aj — A s = (mod p)}. 

Moreover, we call a map cp : A — > B, where A, B C Z 2 , strictly decreasing if 
(p(a)<a for any a G A. Column t of the plane Z 2 is the subset {(t, k)\k G Z}. 
We shall also consider the weight a(s, t) := (0, . . . , 0, 1, 0, . . . , 0, —1, . . . , 0), 
where 1 is at position s, —1 is at position t and 1 ^ s < t ^ re. 
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Theorem 1.1. Let A € X + (n), 1 ^ i < n and 1 ^ d < p. Then the 
module L n (X) contains a nonzero GL n ~i{K)-high weight vector of weight 
A — da(i,n) if and only if for each subset A of 2)^(i,re) whose points are 
incomparable with respect to <, there exists a strictly decreasing injection 
from A to <£ x (i,n) x {0}. 

The existence of the above mentioned injection can be checked using only 
subsets of Z. In what follows, let -k\ : Z 2 — > Z denote the projection to the 
first component. The set A, as well as 2)^(i,n), contains at most one point 
in each column, since d < p. Therefore, there exists a strictly decreasing in- 
jection from A to £ x (i, n) x {0} if and only if there exists a weakly decreasing 
injection from 7Ti(A) — 1 to £ A (i,n). One only needs to devise an algorithm 
to generate all (maximal) subsets of 2}^(«,n) whose points are incompara- 
ble with respect to <. If d=l then all points of 2)^(i,n) are automatically 
incomparable with respect to < and we recover the criterion of [K] . 

In Section[3j we introduce certain elements (elementary expressions) of the 
hyperalgebra. Their multiplication rules (|3.10j) — f)3.12|) represent the ground 
arithmetics behind our approach. Elementary expressions are similar to 
products of the Carter-Lusztig lowering operators introduced in |CL| . In 
particular, the substitution Ui+i, . . . , u n -\ ^ takes s\ d l M {0,...,0) to 
the product Sf mi Sm im2 ■ ■ ■ S^ fcn , where M = {mi < • • • < m/J is a subset 
of (i..n) and Ski is the operator defined in |CLl 2.9]. The remarkable fact 
is that nothing prevents us to view an elementary expression as a similar 
product in the general situation. This idea leads us to the introduction of the 
algebra of formal operators in Section 16.21 As a ring , is simply a 
(commutative) polynomial algebra over a field. However, it is endowed with 

the operators pi = p\ + + pf and cr/ jm . The operator pi represents 
the left multiplication by E\. The operators ai )Tn serve to determine integral 
elements, which appear in Corollaries 16.121 and 16. 14l These corollaries assert 
that certain elements of F- ^ have images in the hyperalgebra and not just in 
its localization. Another approach to integrality is given in [Kj Lemma 2.4]. 

The algebra of formal operators F^ turns out to be a very useful tool 
to express the behavior of the lowering operators. We define the principal 
object of our study, the operators M {I, J), as elements of F- ^ (see Def- 
inition [6]3]) . The operators T^^I, J) are defined similarly to Kleshchev's 
lowering operators (see (K[ Definition 2.3]) and enjoy similar properties given 
in Lemmas l6.5H6.7l (see [Kj Lemma 2.13] or [Bj Lemma 4.11]). 

To some but not all elements of F^ (including T^ M (I,J)), there cor- 
respond elements of the hyperalgebra (see Section [6.6p . However, we want 
to underline that working directly in the hyperalgebra would be very prob- 
lematic. Another argument in favor of F- ^ is that the raising coefficients 
introduced in Section 2] are already products over blocks and conform with 
the factorization procedure described in Section [6j 

Expectedly, the present paper exploits the geometry of the integer plane 
instead of the geometry of the integer line used in [K] and [B] . The corre- 
sponding constructions are given in Section [5l where we define such universal 
concepts as "interior point" , "boundary point" , "cone" and "snake" . 
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Our main operator T.^ m(I) m the hyperalgebra over a field is defined 
in Section 17.11 This is a two-dimensional analog of Kleshchev's lowering 
operator T r;S (M) denned in (Kj § 3]. Moreover, Kleshchev's operator is a 
special case of our operator corresponding to d = 1 and 1 = 0. Notice that 
M is a subset of the integer line in Kleshchev's operator and is a subset of 
the integer plane in our operator. Generally, the use of 7^ M (I) is very 
similar to the use of Kleshchev's operators. Our approach is especially close 
to [B] and |BKSj . Lemma 16.171 show how to calculate 7^ M (I) without 

resorting to the algebra of formal operators F^ . 

Finally, let us notice how this paper is connected with the author's previ- 
ous paper [Sh]. We have Mx{o} ( ) = T $(M, 1) (see [Shi Definition 3.2 
and Section 6]), which shows that in this paper we recover all the lowering 
operators that were repeatedly applied to in |Shj to obtain GL„_i(if)- 
high weight vectors of L n (X). However, we do not know how to apply the 

operators 7^ M (I) repeatedly as in |Sh] . The main obstacle here is the rais- 
ing coefficients for an arbitrary GL n _i(iT)-high weight vector fu,x- These 
coefficients had a very complex form in [Shi Section 4]. Overcoming this 
obstacle would yield the strongest known to date algorithm for generating 
nonzero GL n _i(i ; C)-high weight vectors in the irreducible module L n (\). 

Finally, notice that the restriction d < p is necessary in Theorem II .11 The 
remark at the end of Section [7.51 gives a simple counterexample. Moreover, 
our construction of the operators jf) M (I) does not include the case where 
M has more than one point in some column. Indeed, the definition of r 
by (17. ID is meaningless in that case. The restriction d < p ensures that the 
set n) contains at most one point in each column and therefore so does 
any set M in Sections 17.31 and 17.51 In all our calculations, we assume that 
d < p and consider dl ■ Ik as an invertible element of K. 



2. Notation and definitions 

2.1. Generalities. Throughout the paper, we fix integers i, n and d such 
that 1 ^ i < n and d ^ 1. We define the following sequences of length n: 
ao := (—1, 0, . . . , 0); at ■= (0, . . . , 0, 1, —1, . . . , 0), where 1 is at position t 
and t = 1, . . . ,n — 1. Thus a(s, t) = a s + • • • + at-i- The elements of Z n 
are ordered as follows: A fx if A — /U is an integral linear combination of 
at with nonnegative coefficients. The descending factorial power x— equals 
x(x — 1) • • • {x — n + 1) if n ^ and equals l/(x + 1) • • • [x — n) if n < 0. 
We refer to [GKP, Chapters 2 and 5] for the definitions and relations for 
descending factorial powers and binomial coefficients. Following the stan- 
dard agreement, we interpret any expression a m as the sequence of length m 
whose every entry is a if this notation does not cause confusion. A formula 
A U B = C will mean A U B = C and A D B = 0. For any sequence A, we 
denote by Xj its jth entry. For a set 5, |5| denotes the cardinality of 5. For 
any condition V, let S-p be 1 if V is true and if V is false. 

Let UT(n) denote the set of integer n x n matrices N such that iV aj & = 
unless a < b and UT^°(n) denote the subset of UT(n) consisting of matrices 



4 



VLADIMIR SHCHIGOLEV 



with nonnegative entries. Let e s j, where 1 ^ s < t ^ n, denote the element 
of UT(n) with 1 at the intersection of row s and column t and elsewhere. 
For a matrix N G UT(n), we denote 

• N t :=^2a=i N a ,t the sum of elements in column t of N, 
where t=l, . . ., n; 

• N s := Ylb=i N s ,b the sum of elements in row s ofN, where s=l, . . ., n; 

• N(k):= J2i< a <k<b<n ^a,bi where k=0, . . ., n (summation over a and b). 
We shall often use the formula 

N(l - 1) - N(l) = N t -N l for any N € £/T(n) and I = 1, . . . , n. (2.1) 

Let A € Z n . For two points x = (xi, x 2 ) and y = (yi, 1/2) of [l..n] x Z, we 
define 



One can easily check that dist A (x, y)+dist\(y, z) = dist A (x, z) and dist A (x, x) 
= 0. This notation allows us to write 

2)^(i, n) = {x G (i..n] x [l..d] | dist A ((i, 0), x) = (mod p)}, 

£ A (i,ra) = {s € (i..n) I dist A ((i,0), (s,0)) = (mod p)}. 

We shall also consider the set 

X$(i, n) := {x £ (i..n\ x [0..d] | dist A ((i, 0), x) = (mod p)}. 

Obviously X£(i,n) = 2)^(i,n) U (£ A (i,n) x {0}) U y, where y = {(n,0)} if 
n — i + Aj — A n = (mod p) and y = otherwise. 

2.2. Multisets. A multiset is an unordered sequence. We shall use for mul- 
tisets the same notation as for sets with the difference that instead of the 
braces {,} we shall use the angle brackets (,). For example, (a,a,b) = 
(a,b,a) + (a,b), {a 3 ,b 2 ) = (a,a,a,b,b), (1, 1, 3) U (1, 3) = (1,1,1,3,3), 
(1,1,1,3) n (1,1,3,3) = (1,1,3), (1,1,1,2,2,2) \ (1,1,2) = (1,2,2) and 
(m 2 \m= -1,0,1) = (0,1,1). 

For a multiset I and a set S, let \I\ S denote the number of elements (taking 
into account their multiplicities) of / that belong to S. Let |/| denote the 
total number of all elements of /. We call |/| the length of /. 

For a multiset / and an element x occurring in it at least once, we denote 
by I x ^ y the multiset obtained from / by replacing exactly one x with y. For 
a multiset I with integer entries, we shall use the equalities 

\I q+1 „ q \(-°°-V = |J|(-°°--t) + 8 t=q+1 if q + 1 occurs in /; 

(2.2) 

\I q -i^q\ {t ~ 1} = \I\ {t ~ 1} - 5 t=q + 5 t =q+i if 3 - 1 occurs in I. 
Let m G Z. For a multiset J = (ji, ■ ■ ■ ,jk) with integer entries, we put 
£ m (J) := (min{ji,m - 1}, . . . ,min{j fc ,m - 1}), 
Km(J) ■= {j s \s = l,..., k, js^m-1). 
For a multiset I = . . . , ij) with integer entries, we put 



dist A (x,y) := y ± - x\ + X. 



Aja + x 2 - J/2- 



C m (I) := 
TZ m (I) := 



(i s | s = 1, . . . ,1, i s ^ m - 1), 
(max{ii, m — 1}, . . . , max{i;, m — 1}). 
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We obviously have |£n(.7)| {m ~ 1} = \Tl m (J)\, |ft m (I)|< m-1 > = \C m (I)\. 
Moreover, for m ^ m! , we have 

£m(£m>(J)) = £m(J), C m (C m ' (I)) = C m (I), 

n m ,{K m {j)) = n m ,(j), n m \n m {i)) = n m '(i), (2.3) 
K m {c m ,(J)) = c m ,(n m (j)), n m (c m '(i)) = c m \iz m (i)). 

Below, we give the table that is meant to help the reader to keep track of 
the calculations in Sections O and [7] . 





x < m — 1 


x = m — 1 


x > m — 1 


C m (JU(x)) 


C m {J) U (X) 


Cm{J) U (x) 


£ m (J) U (m - 1} 


K m (J{J{x)) 


Km(J) 


7£ m (J) U (x) 


n m {j) u (x) 


£ m (/U (x)) 


C rn (I) U (x) 


C m (I) U (x) 


C m {I) 


K m (IU(x}) 


K m (I) U (m - 1) 


7£ m (/) U (z) 


n m {i) u (x) 


l~"m ( Jx— h— *x ) 


m ( <^ ) x — 1 1— *x 


£"m\J)x— It— he 


£ m (J) 


7^"m(Jx— It— >x) 


K m {J) 


7£ m (J) U (x) 


T^m{.J)x—l\-^x 






£ m (J) U (x) 


C m (I) 


T^ m (Ix+l^x) 


lZ m {I) 


"^• OT (-f)a;+lH-»x 


T^ m {I)x+i^x 



2.3. Rings and their quotients. Let iti+i, . . . , u n -i be commutative vari- 
ables. Throughout the paper, we assume that m = 0. We consider the field 
of rational fractions Q' := Q(uj + i, . . . , u n _i) and the Lie algebra Qln>(n) of 
all n x n matrices over Q' with respect to usual commutation. Let il(g/q/ (n)) 
be the universal enveloping algebra of 0Zq/(n). 

Let X S) t denote the n x n matrix with 1 at the intersection of row s and 
column t and elsewhere. We also denote Hi := Xi t i, . . . , H n := X n ^ n . 
Thus we have X s ,t G 0^q'( n ) c ^(fl^Q'( n )) an d 

X s ,tXk,i — XkjX St t = St=kX S} i — 5i =s Xk t t, 

for 1 ^ s,t,k,l ^ n. It follows from this formula that the elements 
Hi, . . . , if n3 Mj+i, . . . , u n -i commute with each other. Let U° denote the sub- 
ring of U(gZq/(n)) generated by these elements. By the Poincare-Birkhoff- 
Witt theorem, U° is generated freely by these variables as a Z-algebra. In 
particular, U° is a unique factorization domain. This fact will often be used 
in this paper. 

We obtain a Z n -grading of the algebra il(fl/q/(n)) if we assume that X s>t 
has weight a(s, t) and the elements of Q' have weight zero. Weights and ho- 
mogeneity of elements of H(0Zq/(n)) will always be understood with respect 
to this grading. 

Choosing the appropriate orderings of elements X s j for the Poincare- 
Birkhoff-Witt theorem, we easily obtain that nonzero elements of U° are 
not zero divisors of il(gZq/(n)). 

We claim that H(flZq/(n)) satisfies the right Ore condition with respect to 
the denominator set U°\{0}: for any s G U°\{0} and a G lt(gZq/(n)), there 
exist t G U° \ {0} and b G il(f(Zq/ (n)) such that at = sb. If a is homogeneous, 
then this fact is obvious and we can take b = a. In the general case, consider 
the representation a = a\ -\ Vdki where a\, ■ ■ ■ , a& are homogeneous. Let 
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ii, . . . , tk be elements of U° \ {0} such that ajtj = saj for j = 1, . . . , k. Then 
a ■ t\ - ■ - tk = s ■ X^=i( a j*i ' ' ' tj-itj+i • ■ ■ ijfc) as required. 

Thus there exists the right ring of quotients U of it(g/q/(n)) with respect 
to W° \ {0}. As far as we work over Z, the ring U is our universal object in 
the sense that we construct all the rings we need as subrings of U. 

Note that U is determined uniquely up to isomorphisms identical on 
il(g^q/(n)) and that U is also the left ring of quotients of il(g/q/(n)) with 
respect to U \ {0}. The algebra U inherits the grading from the grading of 
il(g/q/(ra)) described above. 

2.4. Hyperalgebra over Z'. Consider the polynomial algebra Z' := 
Z [it j+i, . . . , u n -i] and denote by U the Z'-subalgebra of it(g/q/(n)) generated 
by 

X*fJ := ^f^- for integers 1 ^ s, t < n, s / t and r ^ 0; 

:= Xs ' s(X3 ' a ~ 1) r 7 (Xs ' s ~ r+1) for integers 1 < s < n and r ^ 0. 

In this definition, the empty product means the identity of il(gZq/(n)). It 
is convenient to define the above elements as zero if r is a negative integer. 
We shall use the notation E^l := and := X\ s ), where 1 ^ s < 

(r) (r) 

t ^ n, and H s := X s<s , where s = 1, . . . , n. We also put Eg ■= E s s+1 for 

s = 1, . . . , n — 1 and omit the superscript W. 
For any TV 6 UT(n), we define 

F (AT) . = TT #(iV) . = TT 

where F^ a ' b ^ precedes F^j iff b < d or 6 = d and a < c in the first 
product and E^' precedes E^ iff a < c or a = c and b < d in the 
second product. Obviously, = E^ N ' = if iV contains a negative entry. 

Proposition 2.1. Elements ^(^-(fj^', w/iere iV, M 
£ UT^°(n) and r\,...,r n are nonnegative integers form a Q' -space basis 
o/H(flZq/(n)). These elements form a Z' '-module basis ofU. 

Proof. This can be proved similarly to [SU Theorem 2]. See also [CL , 2.1]. 
□ 

The similar result holds for {/. Let U° denote the subfield of U generated 
by U°. 

Lemma 2.2. Any element x £ U is uniquely represented in the form x = 
E N ,MeUT>°(n) FWh nm eM , where H N>M eU°. 

Proof. The possibility of such a representation follows directly from Propo- 
sition [57TJ For any element a £ U° and matrix M £ UT^°(n), there is a 
uniquely determined element au £ U° such that E^ M ^a = omE^ m \ 
Now suppose that 

y FW HNM EM=y FWH NM EW\ (2.4) 
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where Hn,m, Hn,m G U° and only finitely many of them are nonzero. Ob- 
viously, we can choose a £ U° \ {0} so that H^m^m > Hn^mum G ^° for all 
N,M G UT>°(n). Multiplying ((23)) by a on the right and applying Propo- 
sition [2J3 we get Hn : mclm = Hn,mo-m for all iV, M G UT^°(n). Noting 
that oa/ / 0, we get J3jv,Af = i?Ar,M- □ 
Let I + and i + denote the left ideals of U and U respectively generated 

(r) 

by the elements Eg , where r > and s = 1, . . . , n — 1. 
Lemma 2.3. i+ n U = i+. 

Proof. We only need to prove that the left-hand side is contained in right- 
hand side. 

Let x G I + H U. The definition of i + and Lemma [231 yield 



x 



V F (N) H (s,r) E ( M )\ E (r) 

^l^s<n, r>0 \^N,M£UT>°(n) N ' M J s ' 



where H { £m It is a standard calculation to check that each element 

is an integral linear combination of elements E^ M ' of the same 
weight. In particular, it is an integral linear combination of E^ M '^ with 
M' G UT^°(n) \ {0}, since r > 0. Hence we get 

^NeUT>°(n), M'eUT>°(n)\{0} ' 

where Ajvm' G U°. However, x G U and Proposition 12.11 and Lemma 12.21 
imply that each 1/jv,M' belongs to the Z'-submodule of U generated by 
various products (^) ••• (^")- Since for M' G UT^°(n) \ {0} we have 
G /+ j we obtain x G i + . □ 
Let [7°, J7 _ , £i _ '° denote the Z'-subalgebras of U generated by the sets 
\( H A | s = l,...,n,r G Z}, {FjJ |Hs<t^n,rGZ}and | 1 < 

s < t ^ n, rG Z}U {H s \ s = 1, . . . , n} respectively. We also denote by 



IK 



U ,0 the subring of U generated by U and U°. If an element x G IA 
represented in the form stated in Lemma l2.2[ then iJjvo for N G UT^°(n) 
is called the F^ -coefficient of x. 

In what follows, we shall use the abbreviation C(k,l) := I — k + Hj. — Hi. 

2.5. Hyperalgebras over fields. Let K be an arbitrary field. Suppose 
that t : Z' — > X is a ring homomorphism. Let if r denote the field if 
considered as a left Z'-module via the multiplication rule f ■ a = r(f)a, 
where / G Z' and a G if. For x G J7, we denote x r := x (g> lif, which is 
an element of U K T . It follows from Proposition 12.11 that the algebra 
U ®z' K T has if -space basis 

{{F m (^)---(^)EW)®1 K \N,M G UT>°(n), n,...,r n integers ^ o}. 

Therefore, this algebra actually does not depend on r. We denote it by 
Uxin). Note that Uk{ji) is naturally isomorphic to the algebra denoted by 
U{n) in [BKS, §2.1]. The reason we constructed the hyperalgebra Uk(ji) 
in such a way is to obtain the projection x i— ► x r , which unlike Uk{ji) 
depends on r, and to have at our disposal the variables tij+i, . . . , it n -i- These 
variables will be used first to construct elementary expressions in Section [3] 
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and then to constructing lowering operators in Section [6j Assuming this 
projection is weight-preserving and that elements of K have weight zero, we 
obtain the Z™-grading of Uk(ji). 

We denote := E^ <g> 1 K for s = 1, . . . ,n - 1, 2$ := ljr for 
1 < s < t < n, IjW := ® 1 K for A € UT>°(n) and 5£ a := # s g> l K , 
(Ms\ ._ (g, ljf for s = i^..^. Let fJ^(n) and E7^(n) denote the 

AT-subalgebras of Uk{u) generated by the sets j(^ s ) \ s = 1, . . . ,n, r G Z j 
and {3r£~l | 1 ^ s < t ^ n, r G Z} respectively. As usual, the 3^ N ^ -coefficient 
°f EjUet/T>o(n) ^ M) h M , where /i M G ^R-(n), is /ijv- 

3. Elementary expressions 
Let Z = 1, . . . , n — 1 and A G UT{n). One can easily check that 

[4^1= E + i)F (iV - e -' +1 - 



1<S<Z 



+jP (^-e M+1 )|^_^ +1 + 1 _ ^ ^ #,+1,6 J (3.1) 

- £ (Ni +h t + l)F^ N - e ^ +ei +^. 

Suppose that for each A G UT(n), we are given an element Hn G U° 
so that only finitely many of these elements are nonzero. It follows directly 
from dO and dO) that 



N&UT{n) MeUT(n) \l^s<i 

+(^-^+i-M,+M,+i+M(Z-l)-2M(0+M(Z+l))iJM+ e! , +1 ( 3 - 2 ) 



E 

Z+l<t<n 



M l+ht H M+ei t _ et+1 1 J (mod f7 • 
Notice also that 

n-1 

F (iV) has weight - E for any A G UT>°(n). (3.3) 

t=o 

Throughout this section we fix a subset = {mi < • • • < m^} of (i..n) 
and additionally assume mo := i, nik+i := n and Jo := ((i — l) d ) (i.e. Jo is 
the multiset of length d whose every entry is i — 1). We consider a sequence 
of multisets I±, . . . , Ik+i, Ji, • • • , Jfc such that 

(Ml) the entries of I s belong to [m s _i— l..m s ), where s = 2, . . . , k + 1, 
and the entries of Ii belong to [i..mi); 

(M2) the entries of J s belong to [m s — l..m s +i), where s = 1, . . . , k; 

(M3) II^I^- 1 } + |J S | = \I S \ + \J a -i\^ m '-^ for any s = 1, . . . ,k. 
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We define the weight 



k ms+1— 1 

£ £ M+|/ s+ i| ( -°°- t] + |J s ! [t - +oo) )a t . 

s=0 i=m s 



(3.4) 



Lemma 3.1. For s = 0, . . . , ft and t = m s — 1, ... , m s +i — 1, i/te at-coeffi- 
dent of\fl M (h, . . . , I k+1 , Ji, . . . , J k ) is -d + |Z s+1 |C-o°-*] + | J s | M-oo). 

Proof. We obviously need to consider only the case t = m s — 1. Applying 
conditions (Ml) - (M3) , we obtain that for s = 1, . . . , ft, the a ms _i-coefhcient 

-d + |J s |(-oo-rn. s -l] + iJ^K-L.+oo) = |J S | + (J,.^™^ 1 } = 

-d+\I s+1 \{ m *- 1 1 + \J,\ = -d+ \I s+1 ^-°°- ms ~^ + |j s |K-i-+°°) 
and the aj_i-coefhcient of A^^Ji, • • • , Jfc+i, Ji, • • • , Jfc) is = —d+ 

|j 1 |(-oo..i-l] + |j [[i-l..+oo) > n 



Lemma 3.2. Suppose that conditions (M1)-(M3) hold for a sequence 
Ji, . . . , Jfc+i, Ji, . . . , Jfc o/ multisets and a set M = {mi < • • • < m^.}. 
Choose any well-defined sequence of the following list 

(i) Ji, • • • , Jjfc+1, Ji, . . . , Jr- 1, (Jr)l-l^l, Jr+l, ■■■ ,Jk'i 

I\, . . . , -Zj., (Jr+ 

where m r ^ / < m r+ i — 1, ^ r ^ ft and r > for the first 
sequence, 

(h) Ji, . . . , Jfc+i, Ji, . . . , J r -2, (J r — l)m r — 2t—*m r — 1 > 

J r U (m r — 1), J r +i, . . . , 

II, . . . , J r _l, J r U (m r — 1} , If+l , . . . , Jfc+1 ! 



Jl, . . ., J r — l, J r U (tTIj. — l), J) 



r+1 



) • • • ! "Kl 



•A- 



Ix, . . . , l r -i,l r U (m r — 1), (J r+ i) mr( _* mr _i, 

J r +2j • • • j Ir+l) Jli ■ ■ ■ > Jk: 
where 1 ^ r ^ ft and r > 1 /or i/te /zrst sequence, 

(hi) Ji, . . . , Jfc + i, Jl, . . . , Jfc_l, (Jfc)n-2^n-i; 

Ji, • • • , h, Jfc+i U (n — 1), Ji, . . . , Jfc, 

where ft > /or i/ie /irsi sequence, 
(iv) Ji, • • • , J r , £ m (J r+ i),7?. m (J r . + i), J r+ 2, • • • , Jfc+i, 

Jl,...,J r _l, £ m ( J r ) , T£. m ( J r ) , Jr+l , • • • , Jfc , 

where m r < m < m r+ i and ^ r ^ ft. 
Jn cases (i) - (hi) , conditions (Ml) - (M3) hold for the chosen sequence and the 



set M.; M this sequence as its argument equals ^^^(Ji 
Ji, . . . , Jk) + on, where I = m r — 1 in case (ii) and I = n — 1 in case (hi) 



(d) 



, J 



fc+l> 



Jn case (iv) , conditions (Ml) - (M3) hold for the chosen sequence and the set 
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M.U{m}; \r® Mv r m \ with this sequence as its argument equals \- ^ ^(4, • • •, 
4+1, Jl, ■ ■ ■ , Jk)- 

We introduce the following element of 

, k m s+ i-l 

^ n ) iM (ii,...,i fc+ i,ji,...,j fc ):=E{^ (jv) n n (^+iJ S |{*- i >)!x 

s=0 t=m s 

x(C(m s ,t) +Um y-(n+\Js\^+\is + i\t-°°-*) I N G UT ^{ n ) and ( 3 - 5 ) 
FW has weight A^ iA1 (/i, . . . , 4+1, 4, • • • , Jfc)}, 

which we call an elementary expression. 

In order to prove that we have actually obtained an element of we 
must show that d ^ N t + | J s |^ -1 ^ + 4+il < "~°°" t ' ) for m s < t < and 
iV G UT^°(n). Since all the entries of N are nonnegative, by Lemma [3,11 we 
have 

N t ^N(t-l) = d- |JJ[*-i"+°°) - IJ^K- 00 -*- 1 ] < 

' + ' (3.6) 

d-|J s |{*-l}-|J s+1 |(— ■*). 

Choose any I = 1, . . . , n — 1. The aim of the present section is to calculate 
EiS^ M (Ii, 4+1, Ji, . . . , Jfc) modulo U ■ Ei. It follows from f|3.2|> that 

( J l> • • • > A+l, Jl, • • • , Jk) = T (mod U-E{), 

where T G W - ' . Let Hm denote the F^'-coefficient of T, where M G 
UT>°(n) It follows from ([321) and that 

T = rf^« M | M G C/T^°(n) and 

(3 7) 

F( M ) has weight Ag M (4, • • • , 4+1, 4, • • • , J fe ) + 

Therefore, we take any matrix M G UT^°(n) such that F^ M ) has weight 

\>,.m(4, • • • ' 4+1, Jl, • • • , Jfc) + Oil < and calculate Hm- By ([331) and 
Lemma |3.1[ we have 



M(t) = d- |/ s+ i | (- 00 -*] - |J s | [t - +oo) - (5*=, (3.8) 
for s = 0, . . ., k and t = m s —l, . . ., m s +i— 1. Hence similarly to (13.60 we have 

d - (M t + |4| { * -1} + |4+i| ( ~ 00 " t) + <$t=j+i) > (3.9) 

for s = 0, ...,k and i = m s , . . . , m s +i — 1. By (j3.2|) . we have = 
Si + S2 + S3 with Si, 52, S3 calculated below. Inequalities (|3.9p show that 
all exponents of the decreasing factorial powers we consider are nonnegative. 

Case 0: I < i. We have TCm = £1 = S2 = S3 = 0. 

Case 1: m r ^ I < m r+ i — 1 for some ^ r ^ k. We put 

*:= II{( M * + \Js\«- 1} y.(C(m s ,t) + Um y-(M t+ \M^ + \is + ^-- «>) 1 
^ s ^ fe, m s ^t < m s+ i, (s,t) ^ (r,l), (r,l + 1)|. 



RECTANGULAR LOW LEVEL CASE OF MODULAR BRANCHING PROBLEM 11 

By (|3.2p we have 

Si = <i>MKM ; -l+|J r .|{'- 1 })!(C(m r ,0+ ^ 

x(% + 1 + |J,|»)!(CK, I + 1) + v) ^-^ +1 +i + |i r |'" + |/ r+1 |'- ' +1 >) ; 

S 2 = - flj +a - Mj + M l+1 + M(Z - 1) - 2M(Z) + M(Z + 1)) x 
x(M t + |J r |^ 1 >)!(C7(m r ,0+^.) d - (M ' +|J - |{ '' 1}+|J - +l|( '° - 0) x 
x(M i+1 +l+|J r |W)!(C(m r ,f + 1) + MtBr ) ^-(^ + i+i + |J r |m + |/ r+1 |(-oc-, + i) ) ^ 

<? 3 = -<&Af*+ 1 (M i +|J r |{'-n)!(C(m r ,Z)+^^ 

x(M m + |J r |W)!(C(m rj I + 1) + Umr) H% 1+ |J f |^ 1 |'--^») i 
We put 

X : = <J>(M* + \J r \ {l ~ 1} - l)!(C(m n + Um J^i^iW^) 
(% + |Jr| W )!(C(m r , I + 1) + Utnr ) ^^ + i+l^l {,} +l^il ( -°°" ,+1) +i) 

Note that ([211} and (P£BJ) imply 

Mi- M l = M(l - 1) - Af(Z) = -l/r+il^ -00 " 1 - 1 ] - |J r |['- 1 " +0 °)+ 

|/ r+1 1 (-00-J] + I J r |[i-+oo) + J = _|J r |{'-l} + |J r+1 |{0 + 1 > - 1 J r [{'-!}, 

whence M t + |J r | { '~ 1} > M l ^ 0. Moreover (f2~T1) and (pTHI) also imply 

M' +1 = M l+1 - M{1) + M(l + 1) = 

M m + |/ r+1 |(- 0O "1 + | J r |« + 1 - \I r+1 \(-°°~ l+1 \ 

Summing Si, S2, S3 and applying (|3.8p . we obtain 

Ha/ = x(M,(C(m r , Z) + u mr — d + Mi + \J r \^ + |/ r+ i|(-°°- J ))x 
x(M /+1 + |J r .|W + l)+ 

(JT, - fl,+i - M t + M l+1 - \I r+1 \(-°°" 1 -^ - |J r |[<-i--+°°) 
+2|/ r+ i|(- 00 - / ] + 2|J r |[ z -+°°) + 2 - llr+xlC-o""^ 1 ] - |J r |[ / + 1 -+°°))x 
x(M, + I J r \V-^)(M l+1 + I J r |W + 1)- 

(M m +|/ r+1 |(— -'] + | J r |« + i_|/ r+1 |(-oc../+i] )(Mz + 1 j r |{i-i}) x 
x (C(m r , Z + 1) + n mr - d + Af m + | J r |« + | X r+1 1 C- 00 -^ 1 ) + 1)) = 
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x(-\J r \V- l \C{m r , l) + Umr -d + M l + \J r \ {l - 1} + |/ r+ i|(- 00 - / ))x 
x(M m + |J r |« + i) 

+ |/ r+1 |^ +1 >(^ + \ J r \V-^)(C(m r ,l + 1) + u mr - d+ |/ r+1 |(-°°- /+] 
-\J r \{^}^M l MJr\ {l - 1} -mC\m r J)+n ^ 

x(M m + |J r |« + l)!(C(m r , I + 1) + u m J d - W + i+l Jr|W + i + |/ r+1 |(-°- ™>) 

+\i r+1 \v+n HMl + |j r |{^>)!(cK,o + ^) d - (M ' +|Jr|0 ' 1}+|f '- +l|( " 00 "' )) x 

x(M m + |J r |W)!(C(m r ,Z + 1) + Mmr) ^-W +1+ |^|W + |7 r+1 |(-^ +1 ) + i) x 

x(C(m r ,/ + l)+n mr -d + 

Note that for I = i, we have = 0, |J | { '~ 1} = d and l^^- 00 ^) = 0. Since 
Ui = 0, the first summand in the right-hand side of the above formula equals 
zero. Taking into account (13. 7|) . (12. 2p and Lemma 13.21 we obtain 

X, Jl, . . ., Jr-lj (Jr)l-l^l, Jr+1, ■ ■ ■, Jk) ,„ 
+ |^r+l|^ +1 ^5'i^,Al(^l' ' ' '»^' (^r+l)z+lW> • • ^1, • • •, </fc) x 

x (C(m r , Z + 1) + u mr - d + |/ r+ i|(-°°- J l) (mod U-Ei). 

Here and in what follows we assume the rule: a product of zero by any- 
thing not well-defined (e.g. a function having an argument Ji—i^i, where 
| J|i' _1 l = 0) is zero. Moreover, the restriction m(Ii, ■■ ■, Ik+i, J\i- ■ ■■> Jk) 
+ai ^ is unnecessary in (|3.1U|) as well as in (|3.11|) and (|3.12p . 
Case 2: I = m r — 1 for some 1 ^ r ^ k. We put 

*:= II{( M * + \Js\ {t - 1} )KC(m s ,t) + ^J jj^jj^ | 

< s ^ k, m s ^.t < m s+ i, (s,t) ^ (r — 1,1), (r,l + 

By (|3.2p we have 

51 = <$>Mi x 

x(M l+1 +l+\J r \W)\(C(m r ,l + l)+n mr ) d -( M '^ +1+ ' J -' { ' }+ '^ 1 ' ( '°°-' +1) ) , 

5 2 = $(F ; - Hi+i — M[ + M l+1 + M(l - 1) - 2Af (Z) + M(Z + 1)) x 
x(M, + \J r ^})\(C(m r ^,l) + MtBr _ l ) ^(^+l^- 1 | {, - 1 >+l^l ( -"' ) ) x 
x(M m +l+lJ r ]W) !(C (m r ,; + 1) + u ^) ±iMl±l±l±W^^ , 

S 3 = -$M /+1 x 

x(M, + | Jr-il^ICCCm,-!,!) + Vl )tWMl« x 



:(M* +1 + |J r |«)!(C(m r ,Z + l) + 



vd-(M m +|J r |f>+]/r-+l| ( - 00 - J+1 )) 
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We put 

(M m + |J r |«)!( C (m Pl I + 1) + UtWr ) ^W + i+l^|W + |/ f+ i|(—'^ + D , 

Similarly to case 1 we obtain Mi + \ J r _i|^ -1 > > M' ^ from and (ET81) . 
Moreover, (12. ID and (13. 8jl imply 

M l+1 = M l+ x - M(l) + Mil + 1) = 

M m + |/ r [(-°°- J ] + jJ r _ 1 |['-+ °) + 1 - |/ r+1 |(-°°" /+1 ] - |J r |[' +1 - +00 ). 
Summing Si, S2, S3 and applying (j3.8jl . we obtain 

% = l(M!(CK-l,!) + «m r _ 1 " d + M; + iJ.-xl^ 1 } + |J r |(-°°" J >)x 

x(M,+i + |J r |W + l)+ 

(fli - fl, +1 - M t + M l+ i - \I r \(-°°- l ~^ - I J r _x I P-i»+oo)+ 
2 |/ r |(-oo.i] + 2 \ J r-1 (['••+«>) + 2 - |/ r+1 |(-°°- J+1 ] - |J r |[' +1 - +0 °))x 
x(M* + |J r _ 1 |{'- 1 })(Af m + [J r |« + 1)- 

(M m +|/ r |(- 00 - i ] + |J r _ 1 |['--+ 00 )+l-|/ r+1 |(- 00 - J+1 ]-|J r |['+ 1 '-+ 00 ))x 
x(M i + |J r _ 1 |f- 1 })x 

x(C(m r ,Z + 1) + u mr -d + M l+l + |J r |» + |/ r+1 |(-°°--<+i) + 1)) = 

^(-|Jr-i| { ^ 1} (C(m r _ 1 ,0+n mr _ 1 -d + Af i +|J r _ 1 |{'- 1 } + |I r |(- oo -0)x 
x(M m + |J r |« + 1) + (M/ + \J r ^\i l - l })(M l+ i + I J r |W + l)x 
x(C(m r _i,m r ) + u mr _ 1 - n mr + \I r \ - \I r+1 \^ m '-^)+ 
(|i r+ i|(-°°-- m "] - \I r \ - iJ^xl^- 1 ) + |J P |)(M, + |J r -i|^- 1 >)x 

x(n mr -(f+^+il^ 11 ))- 

Applying condition |(M3)| for s = r, we obtain — |/ r | — \ J r -i\^ mr ^ 1 ^ + |J r | = 
— \I r+ i\^ rrir ~ 1 ^ , whence 

H M =x(-|J r _ 1 |^- 2 }(C(m r _ 1 ,0+^ mr _ 1 -d+M / +|J r _ 1 |{'- 1 } + |/ r |(-~- J ))x 

x(M m +|J r |«+l) + (Mi + |J r _ 1 |^- 1 >)(M z+1 + |J r |W + l)x 

x(C(m r -i,m r ) +u mr _ 1 -u mr + \I r \ - |J 7 . + i|^ _1 >)+ 

|/ r+1 |{^>(M z + |J,-i| { ^ 1} )(u mr - d +\I r+1 \^-^)) = 

-\J r -i\^- 2 H{Mi + [Jr-i| { '~ 1} - l)!x 

X + U ^-^ *-a|^>- • ") x 

x(M m + |J P |« + l)!(C(m r , f + 1) + u mr ) d - Jr|^> + i + |/ r+ i|(-°- '">) 
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+*CMj + |J r - 1 |^- 1 >)!(CK- 1 ,0 + ^-0 rf - (Mi+|J '-- l|{ '" 1}+|f '- |( ' o °- 0) x 

x(M m + |J r |W + !)!(%, 1 + 1) + ^ ^Hi+I^FWIWI'--^) x 

x(C(m r _i,m r ) +n mr _ 1 - u mr + |J r | - \I r+1 \^^) 

+ \I r+1 \^H(M l + \J r ^ l - 1 ^)lx 

x (CK, I) + u mr _ x f-MMJr-^-VMirt---- % 

x(M m + |J r |W) !( CKJ + 1) + ^-W +1 +I^'>+IWI ( -- ' +1) +D X 

x(n mr -d+|/ r+1 |{ m ^). 
If / = i then similarly to case 1 the first summand of the right-hand side of the 
above formula equals zero. Taking into account (j3.7l) . (|2.2p and Lemma [372], 
we obtain 

Ern r -lS^ M (h, . . . , ifc+1, Jl, . . . , Jfc) = 

0-m r — l>i 

J\i • • • i Jr— 2, ( Jr— l)m r — 2t-*m r — li 

J r U (m r — 1), Jr+i, . . . , Jfc) 



n, JwC-^l) • • •' -^r-lj I r U(m r 1), J r +i, . . ., /fc+1 



j,n,.M 

Jl, . . ., J r _i, J r U(m, r -l), J r+ i, . . ., J fc )x 
x(C(m r _i,m r )+u mr _ 1 -ii mr +|I r |-|I r+ i|^ m ''" 1 ^) 
+|/ r+1 |^}5f n ) M (7i, . . . , Ir^Jr U (m r - 1), (J r+1 ) 

m r i— >m r — 1 , 

I r+2 , . . • , J r+ i, Ji,..., Jk){um r -d+\I r +i\ {mr ~ 1} ) (mod U-Emr-i 
Case 3: I = n — I. We put 



(3.11) 



^ s ^ A;, m s ^ t < m s +i, (s,i) / (k,n— 1)1. 
By (13. 2|) we have 

S 2 = - H l+ i — Mi + M w + M(l - 1) - 2M(l) + M(l + l))x 
x(M, + |J fc |^- 1 >)!(C(m fc , + llmt ) ^W+|J>l^- 1 >+l^l<—-')) , 

We put 



^-(A/ ! + |J fc |0-i} + |/ fe+1 |(— -0) 



X := cD(M, + IJ^'- 1 } - l)!(C(m fc ,0 + 

Similarly to the previous case we obtain Mi + \ Jk\^ l ~^ > M l ^ 0. Notice 
also that M(l + 1) = 0. Summing Si, S2, S3 and applying (|3.8|) . we obtain 

U M = x[Mi(C(m k ,l) + u mk -d + M t + iJfcl^ 1 } + |4+i|(-°°- J )) 
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+(H l - H l+1 — Mi + M l+1 - \I k+1 |(-oo..l-i] _ | Jjfc |[/-i..+oo) + 

2|J fc+1 |(- 0O " J l + 2| J k \V"+°°) + 2 - d)(Mi + |J fc | { ^ 1} ) 

-(M m + \I k+1 \(-^ + |Jfc| [/ - +oo) + 1 - d)(Mi + \J k \ {l ~ 1} )) = 

x{-\J k \^(C(m k ,l)+u mk -d + M l + \Jk\V- 1 } + IW" 00 " ) 

+ {Mi + |J fc | {/ - 1} )(C(m fc ,/ + 1) +u mk - d+ = 

HJfe| {n ~ 2} <W + l4| {W} -l)!x 

x (C(m fc) + n m J d -W + l^' {i - 1} - 1+ l^' ( — 

d>(M, + iJikl^l^m^O + ^j MM.+I^'-^+IW--" % 

x(C(m k ,l + 1) + u mfe - d+ |4+i|)- 

If / = i then similarly to case 1 the first summand of the right-hand side of the 
above formula equals zero. Taking into account (|3.7|) . (|2.2p and Lemma [3.21 
we obtain 

En-lS^M^ 1 ' • • • ' Ik+l, Jl, ■ ■ ■ , Jfe) = 

On— l>i 

\J k \( n 2 ^S inM (Il, ... , Ik+1, Jl,..., Jk-l, (Jk) n— 2i— >n— 1 J 



(3-12) 

l,n,M \ 

x(C(m k ,n) + u mk -d + |4+i|) (mod [7 • J5 n -i)- 



+ S f„ ) .7V,( I i' • • • ,h, h+i U (n - 1), Ji, . . . , J k )x 



4. Coefficients 

In this section, we fix a subset M. = {m\ < • • • < m k } of (i..n) and put 
mo := i, m k+ i := n, T := {(s,t) | ^ s ^ k,m s < t ^ m s+ i}. 

Let Ji, . . . , Ifc+i, Ji, . . . , J k be a sequence of multisets satisfying condi- 
tions 



(Ml) - (M3) together with the set Ai. Consider the representation 



^fn M^ 1 ' • • • ' Ik+i, Jl,-- - , Jk) — —a\ai — ■ ■ ■ — a n -\Ol n -\. 



The integers ai,...,a n _i are determined by (|3.4fl or alternatively by 
Lemma 13.11 We also put ao := 0. In this section, we find the polynomial 
P^n m(Ii, ■ ■ -, Jfe+i, Ji, • • ■) Jfe) of U°, which is uniquely determined, such that 



I7i( a l) 77i( a n-l) qi d ) (J J 1 J 

P$ >M (h, - - - , Jfe+i, Jl, ■ ■ ■ , Jfe) (mod J+). 
Lemma 4.1. J/ai, . . . , a n —l are nonnegative, then 

(k \ / k rn s+ i 

n»™.^^ n n w"- 2 "* 
s=0 / \s=0i=m s +l 

~ x .d+l-l/.+i^- 00 --'- 1 ] 



(4.1) 



(4.2) 



X (C(m s ,t)+Um s )- 



E 



(C(m s ,t)+M ms -|J s |[*-+°°))---(C , (m s ,t)+« ms -|J s |[ t - 1 --+°°)-g) 
<?=0 
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Otherwise Pj^ )M (h, 4+1, 4, ■ ■ ■ , Jk) = 0. 

Proof. If a* < for some t, then s\ d ^ M (h, 4+1, Ji, . . . , 4) =0. There- 
fore, further we consider only the case where a\, . . . , a n _i are nonnegative. 
We apply induction on the weight A^ ^ ^n(-^i) ■ ■ • ; 4+i> 4, • • • > 4), assuming 
that for greater weights the assertion of the lemma holds. 

Case 1: A^ ^(4, • • • , 4+1; 4, • • • , 4) = 0. In (|3.5p . the summation 
parameter iV can be only the zero matrix. Hence 



(k m s +i-l n 
n n \ j *\ {t ~ i}] 
s=0 t=m a / 
/ k m s +i-l \ 

n n (g(m a ,*)+ Mm .) ^i j -i {t - i}+ i^i ( -°" t M 

s=0 t=m 3 / 



(4.3) 



x 



It follows from LemmaOthat d— | Z fl+ i | (-°o. .m a - 1] _ | j s |{m s -i} = | j s |[m s ..+oo)_ 
Therefore, the factor in the second pair of brackets of the right-hand side 
of (|4.3p for t = m s equals u m J-^ — . Let us see what value the same 
factor takes for t = m s +i. By Lemma |3.1| we have 

d- (|4| {ms+1 ~ 1} + |I a +il (-DC " m,+l) ) = 

d- (|J s |[m s+1 -l..+oo) + |j s+l |(-oo..m s+1 -l]) = q 

Hence for t = m s+ \ our factor equals 1. Therefore, (|4.3f) can be rearranged 



as 



S^nM^ 1 ' ' ' • '4+1, 4, . . ., Jfc) = ( 

\s=0 

/ fc m s+ i 

x M] II !4| { ^ 2} !(C(m s ,t)+n m J 



\s=0 / 
k rn s+ i ■> 

rf-(|J B |^- 1 >+|/»+lK- 00 " t - 1 ]) 



\s=0 t=m s +l 

If m s + 1 t ^ m s +i then applying Lemma 13.14 we easily obtain 

(C(m„ t) + . ms ^l^'-^lW---"') = 

(C(m s ,t)+Um s )- 



(C(m s ,t)+u ms -\J s \l t --+°°y)---(C(rn s ,t)+u ms -\J s \l t - 1 --+°°))- 



It remains to notice that we may assume q = in (|4.2p . 

Case 2: Xf^ M (I\, . . . , 4+1, 4, • • • , 4) < 0. Let i be the greatest integer 
of 1, ... j n — 1 such that ai > 0. We have i ^ I and 

iri(ai) p (a n _i) Q (d) / r r j j, 

(4.4) 

-—£j x •••4_i 4 •^i i(n)jM (ii,...,j fc+ i, Ji,..., Jk)- 
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Case 2.1: m r ^ I < m r+ \ — 1 for some ^ r ^ k. Applying (|3. 10[) to the 
last factor of the right-hand side of (|4.4p . we obtain 

H^M^ 1 ' ' ' '' Jfe+l! Jl ' ' ' •' Jfc ^ = ^ ( _ ^>»l J r\ {l ~ X}p i%M^ • • • ' J fe+1' 
Jl, . . . , J r _i, (J r );_i^i, J r +1, • • • , </fc) + 

l^r+ll^^-^n.A*^ 1 ' ■ ■ - ,Ir, (Ir+l)l+lh+l, 4+2, ■ ■ -,-4+l> 4, • • •, «4)x 

x(C(m r ,Z + 1) + u mr - d + [/ r +i| (_01 



Since a; +1 = 0, by Lemma [37X1 we have < o/ = a; — a; +1 = |I r+ i|^ +1 J' 
|J r |«. Hence |/ r+ i|^ +1 > > 0. We put 

x (n^ ' j ' |[ro '" +oo) ) X 

x n?v (c(m.,t) +Mm , r i -'^' ( - oo - t - 11 

111 (C(m s ,i)+« ms -|J s |[*-+°°))-(C(m s ,t)+« ms -|J s |[ t - 1 -+-»)- 9 ) A 



|J. + i|i*- 1 > Uat_! 
<H-2— at-i+g / V q 



(flt_! - flt)« | (s, i) G T \ {(r, Z + 1), (r, I + 2)} 



Considering separately the cases m r < I and m r = I, applying the induc- 
tion hypothesis, the equality ai + \ = and (|4.5|) . we obtain 

P$ M (h, ■ ■ ■ ,4+1, Jl, • • • , J k ) = -*| Jr\ {l ' 1} K\ Jr\ {l} + 

x(C7(m r ,/)+n mr -|J r |[ / - +0 °))x 

/+°° j, ,1, l(-tX3..!l 

x / ^ (C(m r ,l+l)+u mr ) d+1 ] ' r + l] 



(C(m r ,l+l)+u mr -\J r \i l + 1 --+™))--iC(m r ,l+l)+u mr -\J r p--+™)-l-q) 
Kq=0 



x(aJ^ + l +? )(V)^-^) £ 



(C(m r ,«+2)+M mr -|J r |l i + 2 --+°°))-(C*(m r ,/+2)+« mr -|J r .|l i + 1 --+°°)) V ^l" 1 

+$|J r |^- 1 >!|J r |«!|/ r . + i|^+ 1 >: 



x 



(C(m r ,£+l)+w mr )- 



d+i-|i r+ i|(-°°' J ] 

X 



X ( (C(m r ,/+l)+« mr H^r|l i + i: -+ 00 ')---(C(m r ,«+l)+-u mr -|Jr|l i -'+° '-q) 
\?=0 



1,-1 ir i (— oo. .i+il 
(C(m r ,l+2)+u mr ) d+l /|7,. + 1 |{'+i}-l 



(C(m r ,«+2)+n m ,.-|J r |[ i + 2 --+°°))-(C*(m r ,Z+2)+« mr -|J r .|[ i + 1 --+°°)) V <*i-l 

To see that this formula holds in the cases |J r |{*-i} = and I = i, we notice 
that ai_i — ai = |/ r +i|W by Lemma 13.11 in the former case and r = 0, 
u m r = 0, |J r |['" + °°) = in the latter case. Therefore, the first summand of 
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the right-hand side of the above formula equals zero in either of these cases. 
We put 



X := $\J r \V-V\\J r \W\x 



d+i-\i r+1 \(-°°-n 



x (C(m r ,l+l)+u mr ). 

{C(m r ,l+l)+u mr -\J r \\ l + l --+^)y-iC(m r ,l+l)+u mr -\J r \\- l --+°°)) 

(C(m r ,l+2)+u mr ) d+1 -^+ l]( ~ aa " l+1] 

(C(m r ,l+2)+u mr -\J r \l l + 2 --+^))--iC(m r ,l+2)+u mr -\J r \l'+ l --+^))- 

Then we have 



P%,M(h,. .,I k+ l,Jl,.., J k )=-X(\J r \W + 1) ( J2 (J^l q 

,q=l 



(4.6) 



a,-l\ (C(m r ,l)+u mr -\J r \i l -+°°))(H l -H l+1 )!L± I ( \I r +i\ {l+1} \ 
3-1 ) (C(m r ,m)+« mr -|J r |['"+°°)-l)2. J\ H-l ) 

(+oo 
E(^i'X) (V 
q=0 

x {Ht-H^i \ /|/ r+1 |0+i}-l 

{C(m r ,l+l)+u mr -\J r \\i-+^)-l)l J \ a,-l 

Noting that C\m r J)+u mr -\J r \^ + ^=(C(m r J + l)+u mr -\J r \V-- +00 ^-q)- 
((Hi — Hi + i) — q+T), we obtain 



|Ir+l]« \ (ai-A {C{m r ,l)+u mr -\J r \\ l -+^){H l ~H l+1 )^± 
-ai+q ) \ q-l ) (C(m r ,l+l)+u mr -| Jr | l l -+°°)-l)?L 

3=1 

+oo 



3=1 

+oo 



V 9- 1 y (C(m r ,i+l)+ Mmr -|J r |['-+°°)-l)2^i 

nl {i} V a '- lN l (Hi~H l+1 )l = 

- ai +q)\ q-l J (C{m r ,l+l)+u mr -\,J r \\ l -+~>)-l)l 

W/ Vaj-lW I^H-iK V"'- 1 ^ (Ht-H^- 

Z^yyai-i-CH+q+l J\ q J \ai-i-ai+q J\ q-l J J (C(m r ,l+l)+u mr -\ J r \ l '" +oo) -1)^ ' 
<?=0 

Substituting this back to (|4.6p (the first pair of big brackets), we obtain 

+oo / 

\(<H-l\(\I r+ l\V+V 



g=0 
+oo 



*£ [-(l'r| W + l)(j£ 



a i+9+l / V 1 )\ a l~ 1 



q=0 



+i^ 1 i ( ' +ii (^s„)(-')( i '-r 1 ''- 1 

{Hi-H l+1 )3- \ 

X (C(m r ,l+l)+u mr -\J r \l'-+°°)-l)S. I ' 
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Since | 4 |«+l=|4+i| { ' +1} -a*+l, the expression in the square brackets is 



'V \ai-i-ai+q+l J \ q J ^ \ a^-ai+q J \ q-1 

V ai - i ~ ai+1+,? / V i ) J V a < / V a! - i_ai+<? / V 9 / V a ;~ a ;+i 

which immediately yields the required expression. 

Case 2.2: I = m r — 1 for some 1 ^ r ^ k. Applying ([3. lip to the last 
factor of the right-hand side of (|4.4p . we obtain 

^.ImC^ 1 ' • • • ' ^i' ■ ■ ■ ' J k) = 

- y-Smr-l>i\Jr-l\ {mr ~ 2} Pi d n M (Il, 4+1, 

Jl, . . . , J r -2, (4-l)m r -2^m r -l, Jr U (m r — 1), 4+1, . . . , Jfc) 

1 A4 (^i ' • • • ' 4-1 , 4 U (m r - 1) , 4+1 , ■ • • , 4+1 , 

' ' (4.7) 

Ji,... , 4_i, 4 u (m r — l), 4+1, • • • , 4) x 
x(C(m r _i,m r ) + % r _, -w mr + |4| - |4+i| {mr ~ 1} ) 
+|4+i| {nv} fff M (4, • • • , 4-i,4 u (m r - l), (4+1) m r i— >m r — 1 , 

4+2, • • • , 4+1, 4, • • • , Jk)(u mr — d + [4+1 |"t m i !} 



Since a mr = 0, by Lemma |3~T| we have |4+i|^ mr ^ = a™ r -i + |«4|^ m *' ^ > 0. 
We put 



Vs=0 



X 



EE 



(C(m a ,t)+tt ma )■ 



d+l-\l 3+1 



(-oo..t — 1] 



(C(m s ,t)+« ma -|J a |[*-+°°))---(C(m 3 ,t)+« ma -|J a |[ t - 1 --+°°)-g) 
<?=0 



(Ht-t-Ht)! | ( a , t) G r \ {(r-1, m r ), (r, m r +l)} } . 



Considering separately the cases m r _i < m r — 1 and m r _i = m r — 1, applying 
the induction hypothesis, the equality a mr = and (|4.7p . we obtain 

P$ ;M (h, . . . , 4+i, 4, • • • , 4) = -*|4-i| {r ^- 2} !(| Jr| {m -- 1} + l)!x 
x(C{m r -um r - l)+u mr _ 1 - |4-i| [ " v - 1 " +oo) )x 

(C(m r _i,m0+Mm r _ 1 -|Jr-l|[ m ^- + ~))---(C'(m r _i,m r )+«m r _ 1 -|Jr-l|[ m ''- 1 --+ oo )-l-9) 

9=0 
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am r -2-a mr _i+l+g y \^ <? 

(C(m r ,m r + 1)+Um r ) + ' r+1 ' /|/ r+ i|{»"r} 

(C{m r ,m r +l)+u mr -\Jr\ [mr+1 -- +x) )--\C{m r ,m, r +l)+Urn r -\Jr\ [mr -- +oc) ) V am r -i-l 

+$|J r _i|{ m '- 2 >!(|J r |{"^- 1 > + l)!x 

x(C(m r _i,m r ) +n mr _ 1 - u mr + \I r \ - |I r+ i|{ m '- 1 })x 

/ \ ^ ((7(m r _i,m r )+« mr _ 1 ) LL1 

X \ 2^ (C(m r „i,m0+«m r -i-|^-l|[ mr -- + ° o ))---(C*(m r ._i,m r .)+Mm r _ 1 -|Jr-i|[ m ''- 1 -'+ oo '-9) 
\9=0 

V a mr _2— a mr -i+l+q I \ q \ m r i m r j i 



X 



(C(mr,m r +l)+u mr ) d+1 |Jr+l|( °°" mrl / |/r+i| {mi -> 



(C(m r ,m r +l)+« mr -|J r |l m '-+ 1 '-+°°))---(C , (m r ,m r +l)+M mr -|J r |l m '-'-+ °)) \ a mr _i-l 



+$|/ r+1 |{ m ^|Jr-i| {mr " 2} !|«/r| {m '- 1} !(n mr -d + |/ r+ i|^ m '- 1 >)x 

\ -» (C(m r ^ 1 ,m r )+Um r _ 1 ) LJJ 

2^ (C(m r _i,mO+"m r _ 1 -|Jr-l| lmr -- +ro) )---(C(m r _i,m r )+u mr ._ 1 -|J r _i|l^- 1 --+°°)- ? ) 
q=0 



X 



^a mi ._ 2 -a mr _ 1+ l+g^ q ){*m r -l ^mj" x 



J ii It I ( — oo . .m r 1 

(C , (m r! m r +1)+ Mmr ) d+1 |J '- +l1 / |/ r+1 |{™-}-l 



(C(m r ,m r +l)+« mr -|J r |[ m ''+ 1 '-+ °))---(C(m r ,m r +l)+u mr -|J r |[ m '-'-+°°)) V "mr-1-1 

To see that this formula holds in the cases \ J r -i\^ mr ~ 2 ^ = and m r — l = i, 
notice that a mr _2 — a mr -i = |/ r -|''- mr " -1 ^ by Lemma [37X1 in the former case 
and r = 1, u rrir _ l = 0, | J r -i |[ m *-~ 1 "+° ) = in the latter case. Therefore, 
the first summand of the right-hand side of the above formula equals zero 
in either of these cases. We put 

X := ^iJ^il^-^ll^l^-^lx 

(C(m r - 1 ,m r )+u mr _ 1 ) d+1 -^\ ( ~ 00 - mr ~ 1] 

(C(m r _l,m r )+« mr _ 1 -|J r _i|[ m r..+oo))...( C ( mr _ limr ) +nm ^_ i _|J r _ 1 |[m r -l.. + cx ; )) X 

,r<( ,-n, ,d+i-|fr+ll ( ~°°" mr] 
{C(m r ,m r +1)+Um r ) + 

(C(m r ,m r +l)+M mr -|J r |['"'-+l--+°o))...(C(m r ,m r +l)+n mr -|J r .|[ m '---+ 00 )) - 

Noting that by Lemma 13. II 

(w 1 - i| «)( l t: f :') = ww(^!r') =-,-. W). 

we obtain 

(+oo 
Z.^ \a mr -2-a mr -i+q J \ q-1 J (4-8) 
9=1 

^ (C(m r -i,m r -l)+ MmT ,_ 1 -|J r ^i|'" 1 '-~ 1 --+ °))(H mr ,i-H mr )^ ^ ^|/r+i| {m ''> 



(C(m r _l,m r )+Wm r _ 1 -|J'r-l|[ m '- 1 -+ 00 '-l)a I V a mr-l 
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/+oo 

J-n YlX^f \Ir\ {mr - 1} +l \(a mr - 1 -l 

\q=0 

x (H mr -i-H ror )3L Ul / '-+ 1 l {mr> 

^ (C(m r ._i,m r )+« mr _ 1 -|J 7 ._i|[™'- 1 - +°°)-l)2. H a mr _! 

Noting thatC(m r _i,m r .-l) + n mr _ 1 -|J r _i|[ m '- 1 -+ 00 ) = -{B mr _ x -H mr - 
q + 1) + (C(m r _i,m r ) + - | J r _i|[ mr_1 - +0 °) - q), we obtain similarly 

to case 2.1 that 

pSm (h,...Jk + i,Ju---,J k ) = a mr ^x( i w i { ;;> ' 



+ 00 



Z-/ V \ am J'-2- a m r -l+<jy V <? _1 / \am r -2-Om r -l+l+9/ \ 9 / 
9=0 

|/ r |{mr-l} +1 \/a,n r -l-l\\ {H mr ^-H mr )± 

o mr _2-am r -l+l+?/ V, <? / / (C*(m r _i,m r )+n mr _ 1 -|J r _i|l m '-- 1 -- + ° )-l)5. 

■a mr ^x(\^\ {mr} 



^771] 1 

+ 00 



x Vf l^l {mr_1> U-W-A (Hm r -i-H mr )a 

Z^ Va mr _ 2 -a, ni ._i+gy \ « / (C(rJV-l,m r )+« m „_ : , -| J r -i\l m r- 1 ~+° a ')-l)Z ' 
9=0 

which is what the lemma claims. 

Case 2.3: I = n — 1 . Applying (|3.12p to the last factor of the right-hand 
side of (|4.4|) . we obtain 

PiSHuih, ■ ■ • 4, ■ ■ ■ , 4) = ^(-^-i>^l4l {n - 2} x 



X ^,n,A^(4, • • • i 4+1, 4, • • • , 4-1, (4)n-2h 



►71— 1 y 



+^ ( 5m(4, ■ • • , 4,4+1 u <n - 1), 4, . . . , 4): 
x(C(m k ,n) +u mk -d + |4+i|)Y 
We put 



(4.9) 



k m 3+ i \ / k 

-2}, TT7, l^l [ms - +00) 



in* 



•-iin n n w"- 2,] n 

\s=0t=m s +l / \s=0 / 

f +°° ji, I r i(-OO..t-l] 

11 ) Z-> (C(m s ,t)+« ms -|J s |[*-+°°))---(C(m s ,t)+«m s -|J s |l t - 1 --+° c )-9) 

U=o 

x (l^ltl) ( V ) - W I (*> *) e r \ {(*, n)}l . 



ot-2— at-1+9 / \ 9 

Considering separately the cases < n — 1 and rrik = n — 1, applying 
the induction hypothesis and (14. 9p . we obtain 

P$ M {h, ■ ■ ■ , 4+1, 4, • • • , 4) = -$(C(m fc , n - 1) + u mfe - |4| [n_1 " +oc) ) x 

/+OO J_i_i_lr. i(-oo..n-l] 

x f (C(m fc ,n)+« m J± tl-l'^ _ 



I Z-> (C(m fc ,n)+« mfc -|J fc |l«--+-))---(C(m ft ,n)+ ?lmfc -|J fc |l™- 1 --+-)-l- (? ) 
\g=0 

Kj^,:u)(% i - i )(^- if ») £ ') 
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+$(C(m k ,n) + u mk -d+ |4+i|)x 

, it i( — 00..71— 1] 

(C(m k ,n)+u mk ) 1 k + l1 

(C(m fc ,n)+« mfc -|J fc |[»--+°=))---(C(m fc ,n)+n mfe -|J fe |[™- 1 -+°°)-q) 

q=0 




We put 



_it i( — oo..n — 1] 

^ ^ (C(m fc ,n)+» mfc ) + 1 

" (C(m fe ,n)+ Umfc -|J fc |["--+°°))---(C(m fc ,n)+u mfc -|J fe |["- 1 --+ °)) ' 

We have 

/+oo 

J ' " " " ' fe+1 ' J h) - A \Z^\a n -i-a n -i+q)\ q-l 

\q=l 



X 



(C(m k ,n-l)+u m ,-\J k \^- 1 -+^)(H n - 1 -H n )^±- 



+ 



X E 

\g=0 



(G(m fe ,rO+«m ) H J kl [ "~ 1 '' +oo) - 1 ) 1 

\ fon-i-A {H n -i—H n )3- 

„-2-a„_i+l+g^ q J (C(m k ,n)+u mk -\J k \l n - 1 ~ + °°'>-l)Z 



The right-hand side of this formula is exactly the right hand side of (14.8)) for 
r = k + 1 1 
we obtain 



r = k + 1 without the factor a mr ._i( l /r+1 ^ r} ). Hence similarly to case 2.2 

* 1 ' 



•^nUtC 7 " 1 ' • • • ' • • • > «^fe) - 

+oo 

l^ + ll^"^ V a «-^ (ffn-l-ffn)g 

^ V a «-2- a "-l+'?; V « / (C(m fc ,n)+u m -|J ft |["-i--+°°)-l)£ ' 

<?=0 

which is what the lemma claims. □ 
Remark. In case 2, we have o n _i ^ d and, therefore, | Jo|^ _1 -M/a n -i! = 
d\/a n -\\ is an integer. Moreover in (|4.2j) the summation actually goes over 
q = 0, .. . ,at_i = d- |J s+1 |( -DO "* -1 ] - |J s |[ t_1 " +tX) ). This shows that in dO) 
the denominator always divides the numerator in each nonzero summand. 

Hence Pi^ M (h, h+i, Ji, ■ ■ ■ , *4) 6 U° as claimed. 

5. Geometry of integer plane 

5.1. Decreasing injections. Let X be a finite set with a nonstrict partial 
order =<!. We put cone(x) := {y € X \ y x} for any i£l and cone(S') : = 
LLes cone ( x ) f° r an y Scl. A map ip : A^> B, where A,BcX, is called 
weakly decreasing if ?/>(x) =^ x for any x £ A. 

Proposition 5.1. Let A,B C X. The following conditions are equivalent: 

(i) there exists a weakly decreasing injection from A to B; 

(ii) for any S C X, there holds | cone(S') fl A\ ^ | cone(5) fl B\; 

(iii) for any S C A , there holds | cone(5) fl A\ ^ | cone(5) fl B\. 



Proof, (i) => (ii) Let tp : A — > 1? be a weakly decreasing injection. Take any 
Scl. Let y G cone(S') n A. By definition V'(y) £ B. On the other hand, 
there exists x € S 1 such that y =<! x. However ^(y) =^ y. Hence ip(y) =4 x and 
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tp(y) G cone(<5). We have proved ip(cone(S) D A) C cone(5) n B. Since V> is 



an injection, we obtain (ii) 



(ii) (iii) is obvious. 

(iii) =$ (i) Let A = {ax, . . . , a^}, where oi, . . . , a^- are mutually distinct. 
Suppose that for some k = 1,...,N, we have constructed a weakly de- 
creasing injection tpk-i '■ { a i, ■ ■ ■ , ojfe-l} — * -B- We construct the sequence 
S-i, So, . . . , S m of subsets of {ai,...,a&} as follows. We put 5_i := 0. 
Suppose that the subsets S-i, Sq, ■ ■ ■ , S q have already been constructed, 
where q ^ — 1. If cone(5 g ) contains an element of 5 distinct from ip k -i(ai), 
. . . , ^jfe_i(ofc_i), then we put m := q and stop. Otherwise, we put 

:= {a k } U {a; | / = 1, . . . , k - 1 and Vfc-i(aj) G cone(5 g )}. 

If a 6 5 g \ {afc} then ^fc-i(a) G cone(a) C cone(5 g ), since V'fc-i is weakly 
decreasing. Hence a G S^+i- Therefore, 5 g C S^+i (ofc G "S^+i as g + 1 ^ 0). 
Moreover, if q ^ then by hypothesis we have 

\Sq \ i a k}\ < \S q \ ^ | cone(5 g ) n A\ < | cone(5 g ) n B\ 

Thus ipk^i(Sg \ {a k }) C cone(Sg) D -B. Therefore, there exists some b G 
(cone(5'g) nB) \ ipk-i(S q \ {ak})- Since we consider the case q < m, we 
have 6 = tpk-i(a), where a G {ai, . . . , a^-i} and a £ S q . We see that 
a G S^+i \ S q . Therefore, the process of constructing the sets Sq, Si, ■ ■ ■ will 
eventually terminate. Note that m ^ and Sq = {a k }. 

Let b m be an element of cone(5 m ) distinct from ^_x(ai), . . . , V'A:— l («fc— i)- 
We are going to construct inductively the elements b m ,...,bo of B and the 
elements a' m , . . . , a' of {a±, . . . , a&} so that 

(a) bi G cone(aj) \ cone(S/_i) for any I = 0, . . . , m; 

(b) a'i € Si\ Si-i for any ! = 0,...,m; 

(c) Vfc-i(aJ) = fy-i for any 1 = 1, ...,m. 

If 6 m belonged to cone(S' m _i), then this element would coincide with 
one of the elements Vfc-lX&i), • • • j^fc-ifajfe-i)- Therefore 6 m G cone(5 m ) \ 
cone(5 m _i). We take for a' m an arbitrary element of S m \ S m -\ such that 
b m G cone(a^). 

Now suppose that the elements b m , . . . , b q and a' m , . . . , a' , where m ^ 
q > 0, have already been chosen so that conditions (a) and |(b)| hold for 



I = q, . . . ,m and condition (c) holds for I = q + 1, . . . , m. 

To conform with condition (c) we put b q -\ := ijjk-i(a'). Note that 



a' q G S q \ Sq-i C S q \ {a k } C {ai, . . . , a k -i}- We obtain G cone(S 9 _i) \ 
cone(5 g _2). Now we take for a' q _ 1 an arbitrary element of S q ^i such that 
b q -i G cone(ag_ 1 ). If a' q _ 1 belonged to 5 9 _2, then b q ^\ would belong to 
cone(5 g _2), which is wrong. 

Note that a' Q = a k . Now a weakly decreasing injection ifi k : {a\, . . . , a k } — > 
B is given by 

, . J ^fe_i(a) if a g {o^, . . . , a^}; 
j • \ 6j if a = a\ and Z = 0, . . . , m. 

The injection we need is tpjy. □ 
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5.2. Decreasing injections for Z 2 . Recall that in Section [H we intro- 
duced the strict order < and defined strictly decreasing maps. However, a 
better theory exists for nonstrict versions of these concepts. See, for exam- 
ple, Proposition 15.11 

We introduce the nonstrict partial order ^ on Z 2 as follows: (a, b) (x, y) 
holds if and only if a ^ x and b ^ y. Let A, B C Zi 2 and ip : A — > B be a 
map. We call <~p weakly decreasing if ip(a) ^ a for any a G A For any point 
a G Z 2 , we put cone(a) := {/? G Z 2 | /? ^ a}. For any subset T C Z 2 , we 
put cone(r) := \J a€T cone(a). We also endow Z 2 with the componentwise 
arithmetic operations. For a point a G Z 2 and a subset T C Z 2 , we put 
r±a:={7±a|7G T}. 

Proposition 5.2. Lei A and B be finite subsets of Z 2 . // there is a strictly 
decreasing injection from A to B, then |cone(r)rij4| ^ | cone(r — (1, 1)) nB\ 
for any T C Z 2 . Conversely, if | cone(r) PI A\ ^ | cone(r — (1, 1)) n B\ for 
any T C A, then there exists a strictly decreasing injection form A to B. 

Proof. Clearly, there is a strictly decreasing injection from A to B if and 
only if there is a weakly decreasing injection from A to B + (1, 1). Now the 
result follows from Proposition 15.11 and the obvious formula 



cone(r) n (B + (1, 1)) = (cone(r - (1, 1)) n B) + (1, 1), 
where T is an arbitrary subset of Z 2 . □ 

5.3. Snakes. Let A C Z 2 . A point a G A is called an interior point of A if 
there exists b G A such that a < b. Otherwise o is called a boundary point 
of A. The sets of all interior points of A and of all boundary points of A 
are called the interior of A and the boundary of A respectively. For T C Z 2 , 
we define snake(r) to be the boundary of cone(r). Clearly, snake(r) = 
cone(r) \ cone(r — (1, 1)). 

Proposition 5.3. Any two points of a snake are incomparable with respect 
to < . Conversely, if T is a subset of Z 2 whose points are incomparable with 
respect to <, then T C snake(r). 

Lemma 5.4. Let R = [a..b] x [c.d] be a nonempty rectangle and X C R. 
We put R := [a..b] x {c}, Ri := [a..b] x (c.d] and Y := X D R\. The 
following conditions are equivalent: 

(i) there exists a strictly decreasing injection ip :Y — > X; 

(ii) for any subset A of Y whose points are incomparable with respect 
to <, there is a strictly decreasing injection from A to X PI Rq. 



Proof, (i) => (ii) We endow X with the following nonstrict partial order: x ^ 
y if and only if there are elements zq, . . . , z m , where m ^ 0, of X such that 
y = zq, x = z rn and z s = t/>(z s _i) for any s = 1, . . . , m. For any y G X, let 
V>(y) denote the smallest (w.r.t. ^) element of {x G X \ x ^ y}. Obviously, 
ip(y) G X n Rq and ^(y) = 4>{y') if and only if y and y' comparable with 
respect to 

Let A be a subset of Y whose points are incomparable with respect to 
<. Then distinct points of A are incomparable with respect to ^, since ip 
is strictly decreasing. Hence V'Ia is a required injection. 



RECTANGULAR LOW LEVEL CASE OF MODULAR BRANCHING PROBLEM 25 



(ii) (i) We shall use Proposition 15.21 Let T C Y. We must prove the 



inequality 

| cone(r) n Y\ < | cone(T - (1, 1)) n X\. (5.1) 

We have 

cone(r-(l,l))nX = (cone(r-(i,i))nr) u(cone(r-(l,l))n(Xn J R i 

Since the intersection of cone(r) n Y and cone(r — (1, 1)) n X is cone(r 
(1,1)) H Y, inequality (15. 1|) is equivalent to 



^cone(r) \ cone(r - (1, 1))J n Y < | cone(r - (1, 1)) n(In R )\. (5.2) 
Consider the snake S := snake(r). We have 

cone(r) n (Y n S) = (cone(r) \ cone(r - (1, 1))) n Y. 



The last set is exactly the set of the left-hand side of (15. 2p . Therefore (|5.ip 
is equivalent to 

| cone(r) n (YnS)K [ cone(r - (1, 1)) n (x n R )\, 



which holds in view of Proposition 15.21 and the condition of part (ii) Here 
we should take A := Y n S and apply Proposition 15.31 □ 
Also, we call a map ip : A — > B, where A, B C Z 2 , strictly increasing if 
<p(a) > a for any a E A. 

5.4. Set S. Consider a nonempty subset S of Z 2 having the form 

S = ({a} x [f a ..l a ]) U ({a + 1} x [/ a+ iJ a+ i]) U • • • U ({b} x [f b ..l b ]), (5.3) 

where a ^ 6, / s+1 ^ / s ^ l s+ i ^ Z s for s = a, . . . , b — 1 and / s ^ / s+ 2 for any 
s = a, . . . , b — 2. We say that a point x = (xx,X2) of the strip [a..b] x Z lies 
below S or above 5 if X2 < f Xl or X2 > l Xl respectively. 

Lemma 5.5. Let M C X C [a..b] x Z and y> : M — > X be a strictly 
increasing injection. Suppose that X f]S does not contain points comparable 
with respect to <. Then there is an injection (p$ '■ M — > X \ S such that for 
any x E M we have either x < <ps(x) or fsip^j = x an d x below S. 

Proof. We denote by B and T the set of all points of [a..b] x Z lying below 
S and above S respectively. For x E M, we put ips( x ) '■= x if x E B (1 M 
and (fs( x ) '■= ^(x) otherwise. It remains to show that ips is an injection 
from M to X \ S. 

Take a point x = (x\,X2) E M and denote (2/1,2/2) := y(^)- We have 
yi > xi and 2/2 > x 2 . If x G B then <£>s(x) = i£BnMcBn!cI\S. 

Now suppose x E 5. If we had i^(x) € S, then x and v 9 ( a; ) would be 
points of 1 fl S comparable with respect to <. Hence <~ps( x ) = ¥>(aO ^ <5- 
Suppose that </?(x) E Then 2/2 < ^ f Xl ^ ^2, which is a contradiction. 
Therefore ^(x) = <p(x) £TnX C X\S. 

Finally, let x E T. We have 2/2 > £2 > fci ^ lyi- Hence ips(x) = <p{x) E 

Tnxcx\s. ' □ 

Now let us consider the case where S 1 contains points comparable with 
respect to <. We fix some A E Z n and suppose that 1 ^ a ^ b ^ n. 
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Lemma 5.6. Let x,y G S and x < y. Then x and y belong to adjacent 
columns and 

(A* - h+i) 1 ^^- = (mod dist A (x, y)), 
where x belongs to column t. 

Proof. We denote x = (t,u) and y = (s,v). Since x < y, we have t < s 
and u < v. If x and y did not belong to adjacent columns, we would have 
t + 2 ^ s. By the conditions imposed on S, we get v > u ^ f t ^ Z t+ 2 ^ Z s 
and therefore y £ S, which is a contradiction. 

Thus x = (t, u), y = (t + l,v), ft ^ u ^ ^ and / t+ i ^ t> ^ it+i. Hence we 
get 

< v - u < h+x - ft (5.4) 

By definition, we have At — Af+i = v — u — 1 (mod dist^(x, y)). This equiv- 
alence together with (15. 4p . yield 

(A t - A m )^+i^ = («-«- 1)*«±1=* = (mod dist A (x, y)). 

□ 

5.5. Diagrams. Let k < j and I be a multiset with integer entries of length 
no greater than d. We put 

sjg.(J) — {( t ,h) € Z 2 | fc < t <j &0 < h < d- |/|(-°o-*-i]} , 
njg.(J) : = {(£,/!) G Z 2 | fc < t <j &0 < h< d- |/|(-°°"*]} . 

One can easily see that fijj, ^ (J) is the interior of t!^ 1 - (I) . 

Lemma 5.7. Let T be a subset of(i..n] x [0..d] whose points are incomparable 
with respect to <• Then there exists a multiset L of length no greater than 
d with entries in the interval [i..n) such that V is contained in the boundary 

Proof. We denote by V the set of all points of T not belonging to column 
n. Let t\ < ti < ■ ■ ■ < tk be the numbers of columns that contain at least 
one point of V . We denote hj := max{/i | (tj, h) € T'} for j = 1, . . . , k. Let 
hk+i '■= max{/i | (n,h) € T} U {0}. Since the points of T are incomparable 
with respect to <, we have h\ ^ hi ^ • • • ^ % ^ frfc+i- Now we put 

r / -d—hi j.hi—h,2 Ji2—h.3 j.h k _ 1 —h k ,h k —h k+1 \ 

1 V > C 1 > l 2 ! • • • ! r fc-l ) E fc /• 

This multiset has length d — hk+i ^ d. We put ifc+i := n. 

Take a point 7 = (tj,h) £ T. By definition, we have /i ^ hj. Since 
| 7 |(-oo..%-i] = d-hj, we have 7 G e£2(J). 

It remains to show that 7 ^ f^(/). This is true if 7 belongs to column 
n. Therefore assume that 7 € T'. If h < /ij+i, then 7 < G r and 

we get a contradiction with incomparability of points of T. Hence hj+i ^ h. 

Since = d- h j+1 , we get 7 £ nJJ(i). □ 
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6. Formal factorization of elementary expressions 
6.1. Cutting operators. Let i ^ I < m < n. For t = 1, . . . , n, we put 



We extend a^ m to a ring endomorphism of U , assuming that a^ m acts iden- 
tically on Ui+i, . . . , u n -\. One can check directly that a^ m is an idempotent 
operator on U° and that for 1 ^ q ^ t ^ n, there holds 



We extend a^ m to a Z'-module endomorphism of U ,0 by the rule a^ m (FH)= 
Fa ltTn (H), where F G Z7 - and H G W°. 

Lemma 6.1. An element x G Zi~'° is representable in the form x = 
y(C{l,m) — u m + u{) for some y G if and only if ai >m (x) = 0. 

Proof. We put for brevity a := o-^ m . Clearly, if x = y{C{l,m) — u m + u{) 
for some y G then cr(aj) = 0, since a(C(l,m) — u m + u{) = by (|6.ip . 

Conversely, suppose that <r(x) = 0. Take any matrix M G UT^°(n) and 
let Ti.M be the F^^^-coefficient of x. By our assumption and Lemma 12.21 
o~(TCm) = 0. Let R be a subring of U° generated by Hi, . . . , H m -\, 
H m+ i, . . . , H n and itj+i, . . . , it n -i- We have the representation 



where ho, . . . , h k G i?. Applying <r to this representation, we obtain 

a(h k )a(H m ) k + o-(h k _i)o(H m ) k ~ l + ■■■ + a{h x )a(H m ) + a(h ) = 0. (6.2) 

Consider the ring endomorphism r of U° such that r{H m ) = m — l + Hi — 
Um+ui and r acts identically on the generators Hi,.. ., H m _i, H m+ i, . . -,H n , 
Ui + i, . . . ,ii n _i. We claim that (joo)(Ht) = H t for any t G {1, . . . ,n}\{m}. 
lit <m then (r o a)(H t ) = r(H t ) = H t . Now let t > m. We have 



Thus, we have (r o a) (ho) = ho,. . . ,(r o a)(h k ) = h k . Moreover, a(H m ) = 
m — l + Hi — u m + ui and it does not depend on i7 m . Therefore (ro(j)(F m ) = 
m — I + Hi — u m + . Thus applying r to (|6.2p , we obtain 

h k (m - I + Hi - u m + ui) k + hk-i(m - I + H - u m + ui) k ~ l -\ 

+hi(m - I + Hi — u m + ui) + h = 0. 

We have proved that m — I + Hi — u m + ui is a root of the polynomial 

+ h k -iX k ~ 1 H h hiX + ho G belonging to the ground ring R. 

By the Bezout theorem, this polynomial is divisible by X—m+l—Hi+u m —Ui, 
that is 

h k X k + h k _ x X k - x + • • • + hiX + /i = P(X) • (X - m + Z - fl, + um - ui) 

for some P G Substituting £Z m for X, we obtain 

% = P(H m )-(H m -m + l- Hi + Um-ui) = -P{H m )-(C{l,m)-u m + ui). 



0~l,m(Ht) = 



H t if t < m; 

.fZf + C(Z, in) — u m + ui \i t ^ m. 




(6.1) 



% = Ztfcil^ + hk-iH m 1 + • ■ ■ + hiH m + ho 



(t o a)(H t ) = r(H t + m - I + Hi - H m -u m + u{) 
Ht + m — l + Hi — r(H m ) -u m + ui = H t . 
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It remains to notice that P(H m ) G Li°. □ 
Remark. We shall often use the following simple fact immediately follow- 
ing from (|6.ip : for I < m and q <t, the equalities ai jm (C(q,t) —Ut + u q ) = 
and (l,m) = (q,t) are equivalent. 

Lemma 6.2. Let M = {mi < • • • < m^} be a subset of (i..n), m,Q := i, 
m k+i '■= n an d 111 ■ ■ ■ 1 1k+ii Ji, ■ ■ ■ ,Jk be a sequence of multisets that with M. 
satisfies conditions |(M1)| - |(M3)[ Suppose that ^ r ^ k, m r < m ^ m r+ \ 
and m < n. Then we have 

C r m ri m(5'^ ) _ A/( (/l, . . . , h+1, Jl,--- , Jk)) = 

^^.A/fU-jm}^ 1 ' ' ' ' ' Ir,£ m {Ir+l),TZ m (I r+ i), I r+2 , ■ ■ ■ , h+1, 
Jl , ■ ■ ■ j Jr— 1; £"m\Jr), 7^m\Jr) , Jr+1 , • • • , Jk) 

if m < m r+ i and 

(si n^M^ 1 ' " ' " ' Ik+1, Jl, ■ ■ ■ , Jfc)) — S i n M {Il, . . . , ifc+1, Jl, . . . , Jk) 

if m = m r+ \. 

Proof. First consider the case m r < m < m r+ i. Take an arbitrary matrix 
M G UT>°(n) such that F< M ) has weight AjJ )A< (Ii, . . . , Ji, . . . , J k ) 

and denote by TLm the F( M ) -coefficient of S^ M (Ii, . . . , ife+i, Ji, . . . , Jfe). 
By (J331) and ([EH), we have 



m r ,ml 



(^)=(iLMMr>in:" 1 ^ + i j -i {t - i} ) ,x 



XO-.nr.mfnr 4 " 1 '( M * + |Jr| { *~ 1} )!x 
\ J - J -t=m r 

x (C(m r ,t) + Umr) ^(M t+ |J,.|0^> + |/, +1 |(— )) 

-Llse[0..fe]\{r}lJ-t=m s ; 

x (C(m s , t) + u ms ) ^(M t+ | Js |0- 1 > + |/ s+1 |(—)) 

r m— 1 



■ J - J -t=m r 

x n" 1 " 1 ^ + | J r |{*-i})!(Cr(m,t) + nm ) ^-(M t+ |i^-> + |/.. +1 |'— )) . 

-*-t=m 

To complete the proof in this case, it remains to notice that 
|J r |{<~i} = \C m (J r )\^-^ and iJ^+ijC-^-*) = |jC"»(J P+1 )|(-°°"*) for t < m; 
\Jr\V- 1 } = \TZ m (J r )\^-^ and [J^ilC- 00 "*) = |7e m (I r+ i)|(-°°- i ) for t ^ m 



and apply case (iv) of Lemma 13.21 (the sequence of this case is always well- 
defined). 

In the CELS6 771 — 7Tl r ^,\j it follows frOHl f]3. 5 j) cLIld (|6.ip that (Jjji t ui 

acts 

identically on any F^^-coefficient of s\ d ^ M (I\, . . . , Ik+i, Ji, ■ ■ • , Jk)- d 
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6.2. Factorization. We consider the formal commutative variables 
S^ m ,(I, J), where i ^ m < m! ^ n and 

(Fl) / is a multiset with entries in [m — l..m') if m > i and with entries 

in [i..m') if m = i; 
(F2) J is a multiset with entries in [m— l..m') if m > i and J = {(i — 

if m = i. 

These variables are called formal operators. Let denote the free 
commutative algebra (polynomial algebra) over the field U° generated by 
all formal operators. This algebra is called the algebra of formal operators. 

A monomial Q ■ f /g, where Q is a product of formal operators, f,g£U° 
and g ^ 0, is called regular if the following conditions hold for Q and g. 

(F3) If S { l] m {IJ) and S^ m ,(I, J) occur in Q, then |/|{— ^ + |J| = 

\I\ + |J|{ m-1 >. 

(F4) For any t € Z, there is at most one factor of Q having the form 

S m,m'^ 1 ^)^ wli ere t G [m..m'). 
(F5) g is a product of polynomials C(m, ml) — u m i + u m , where i ^ m < 
m! < n, each in degree at most 1. Moreover, if this polynomial 
occurs in g, then some formal operator <S^ ' «(/, J), where m! ^ m", 
occurs in Q. 

A polynomial of is called regular if it is a sum of regular monomials. 

We are going to introduce the cutting operators for F^ similar to the 
cutting operators of Section 16.11 for which we shall use the same notation 
&m,m'i where i ^ m < m! < n. 

At first, we define the action of 0" m>m ' on the formal operator S^ m „(I, J). 
If m = m and m! < m", then we put 

VmM^m" ( J ' J )) -^m'^'^^iJ)) S^ m „ (K m ' (I),7^ (J)). 

If the intervals [m..m') and [m..m") either do not intersect or coincide, then 
we assume that o- m ^ m i acts on <5^ m „(J, J) identically. In all other cases, we 
define the result of this action to be 0. 

Further, we assume that o- m ,m' acts on U° as defined in Section loTTl We 

define the action of <r mjm / on an element P € F^ as follows. Suppose that 
P is representable as 

i=i 

where Sq are formal operators, fj,gj £ U° and a m ^m'{,dj) 7^ f° r an Y 
j = 1, . . . , I. Then we define 

l 

0~m,m'{.P) = Q~m,m' (S^ ) • • • 0~ m ^ m i (Sf. ) • 0~ m ^ m i (fj) / Om,m! {dj) 
3=1 

It is easy to understand that a m>m i (P) thus defined does not depend on the 
choice of the representation of P. 
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Remark. The operator cr mm i is not applicable to elements of not 
representable in the above form (see the example below). However if a m ,m' is 
applicable to P and P' of F>£ , then a m ,m' (P ± P') = Om,m> (P) ± °~m,m> (P > ) 
and <j mtrn >{PP') = <y m ,m'{P)<y m ,m'{P')- 

A polynomial P € F^ is called integral if it is regular and for any i ^ 
m < ml < n, there holds 



Vm,m> (jP ■ (C(m, m) - u m i + u m )\ 



0. 



In this formula, cr m ,m' is applicable to P • (C(m, m') — u m i + u m ), since the 
denominator of the last polynomial can contain only factors of the form 
C(mo,m' ) — u m / o + u mo , where (mo,m' ) ^ (m,m'). By (|6,1|) . the operator 
&m,m' does not take such polynomials to zero. 

Example. Let i < m < n, I be a multiset with entries in [i..n), Jq = 
{(i - l) d ) and 



p = s™(i,j )- slZ(c m (i), Jo) s^n(n m (i), 0) 



C(i,m) - u m 

This polynomial is integral. Note that o"j )TO is not applicable to P. Indeed, 
suppose that there exists a representation P = Y^j=\ Qj'fj/9ji where Qj are 
products of formal operators, fj,gj 6 U° and (Ji, m (gj) ^ 0. Then g%,...,gi 
are not divisible by C(i, m) — u m in U°. We have 

91 ■ ■ - gi Jo) - s^l(c m (i), Jo) s$ n (n™(i), 0] 

(C(i, m) - Um) Ej=i Qj ' fj9X'" 9j-i 9j+i '~9i- 

The coefficient at <s}„(I, Jo) in the left-hand side equals g\ •••<?/ and is 
not divisible by C(i,m) — u m unlike the entire right-hand side. This is 
a contradiction. 

Finally, we introduce the notion of weight for elements of F^ . We say 
that 

• 5 Sn'( J > J) has weight ET=m\~ d + |/| ( "°°" t] + I Jj [ '" +oo) H; 

• elements of U° have weight 0; 

• if P has weight A and P' has weight //, then PP 1 has weight A + fi; 

• a sum of elements having fixed weight A has weight A. 

• a sum of elements having weight ^0 is called an element of weight ^0. 

6.3. Raising operators. Following the multiplication rules of Section El 
we introduce the operators p^\ pf , pf , p\ 3 \ where i ^ I < n, on the 
algebra of formal operators F^ . 

We assume that all these operators act identically on U°. Take a formal 
operator <5^f , (I, J) and an integer I such that i ^ I < n. We put 



J\ {l - 1} S^(I,J M ) if m^l < 



S^ m ,(I,JU(m-l))ifl = m 
S m,m'( I ' J ) otherwise, 
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P r } (^ m ,(I,J)):=i 



'0 if m ^ I < m! — 1; 



5 mJn'( Ju < m ' " 1), J)(C(m,m')+«m-d+|/|) 

if / = m' — 1 ; 

Sm, m >( I > J ) otherwise, 
'0 if m < / < m' - 1; 

^Sn'^UK- 1 )" 7 )' ifi = "i'-l; 

JU <m - l))(n m -d+|/|{— !>) if l=m-l; 
K^rrA^J) otherwise. 



f|/|0 +1 >S<? m ,(W,, J)(C(m,l+l)+u m -d+\I\(-™ -']), 



5^,(/U(m'-l},J) if/ = m'-l; 
,(d) 



if m— 1 ^ / < m'—l; 



, S m'm'(^ J ) otherwise. 



We extend the operators p[ 2 ' L \ p\ 2 ' R \ P^f 1 to ring endomorphisms of 
F^. Finally we put pf ] := pf> L) - pf> R) . 

Remark. Clearly, each p^\ p^\ p\ 2 ' L \ p\ 2 ' R ^ acts on a formal operator 
S^,(I,J) identically unless m — 1 ^ I ^ m' — 1. The reader should 



(2) 

be careful with p) , keeping in mind that it is not a ring endomorphism. 
However, if both p[ 2 ' L ^ and pf^ act identically on x G , then p^ 2 \xy) = 
xpf\y) for any y G i^J. 

6.4. Operators T^ M (I,J). We are going to define the principal object 

of our study, the elements Tjf^ M (I,J) of F-^ , where i ^ k < j ^ n; 
A4 C (k..j); I and J are multisets with entries in [k — if k > i; I is a 
multiset with entries in and J = ((i — l) d ) if k = i. 



Definition 6.3. 



3. We put TW (I, J) := S K k a J(I, J). For M^0,we put 



- .c(«0 



T 



xTi% X{m} (K m (I),K m (J)))/(C(k,m)-u m + u k ), 
where m = min A4 . 
Lemma 6.4. 



where k < mo < min M. U {j}. 



(i) T k d j,M(^ J ) is re 9ular. 
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(iii) T^ M (I,J) is integral. 



Proof, (i) The result follows from the following observation, which can easily 
be proved by induction on \M\: T^ d j M (I, J) is an integral linear combination 
of monomials Q/g such that 



G = nl =0 5 ^>- s+1 ( £nis+1 (^ Ws «)»^ +1 (^(J))); 

g is a product of polynomials of the form C(m s ,m') — u m i + u ma , 
where s = 0, . . . , c, m s < m' ^ m s+ \ and m' £ .M, each occurring 
in g in degree at most one, 



where c ^ 0, k = uiq < m\ < • • • < m c < m c+ i = j and mi, . . . , m c € «M. 
To see this, it suffices to apply Definition 16.31 and (|2.3p . 



(ii) Induction on \M\. The case .A/f = is obvious. 



Now let M ^ and suppose that the result is true for sets of smaller 
cardinality. We put m := min.M. Applying Definition 16.31 (|6.ip and the 
inductive hypothesis, we obtain 



( Tk,m {1'k ( j > M( I ' J ) 



0~k, 



T, 



(d) 



,m \-*k,j,M\{™y 



{I, J)) -Ok,rn {S<$ n {C n {I), C m (J)) )X 



xT 



(d) 



m ,j,M\{m\ 



(n m (i),K m (j)) 



j (C(m , in) -u m + u mo ) 



-(d) 



-<io^ mo ( £m ( / ))' £ ™o(^(J)))5^l m (7e-o(£™(/)),^ m j/: m (j)))x 



(d) 



/(C(m ,m) - u m + u mo ) 



k,mo 



(C m °(I),£ mo (J)) 



T 



(d) 



mo ,j,M\{m} 



{n™*{i),Km {J)) 



S)nl m {n m °{C m {I)),1lm {Cm{J)))* 



xT 1 



{d) 



iM\{m } (K m (I),Km(J)) 



/(C(m ,m) - U m + Um ) 



5 (<0 

A;, mo 



(C m °(I),£ mo (J)) 



T 



(d) 



mo j',.M\{m} 
-SfiU (£ m (^ mo (/) ) , £m (^mo ( J) )) X 

*T^] MX{m} {n m {K m »{I)),Km{Km {J))) 



{n m °{i),n mo {J)) 



/ (C(m , m) -u m + u mo ) 



(iii) Induction on |«M|. The case = is obvious. Now let 7^ and 
suppose that the result is true for sets of smaller cardinality. We put m := 
minA4. Applying the inductive hypothesis and the remark of Section [6.21 
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we obtain 

v mo ,m' ( T k d j, m( J ' J ) ' (C(m ,m' Q ) - u m r o + u mo )) = 
°~m , m ' A^\{ m }( / > J ) ■ (C(mo, m ) - + u mo )j - 

^mo, m 'o(41(^ m ( / )^m(' 7 )))x 

X ^m ,m^( / ?lj iA1 \ {m} (^ m W,^m(^)) ' (C(m ,m' ) - U mJ) + U mo )j I 
°m ,m' (C( fe > m ) - u m + = 

for any mo, m such that i ^ mo < m' < n and (mo,m ) ^ (k,m). The 
only remaining equality 

follows immediately from part (ii) of the current lemma. □ 
Lemma 6.5. Let i ^ k < j ^ n and k ^ I < j. Then we have 

Ifi<k then ^(^(J, J)) = T$ M (I, JU(k- 1». 

Proof. We apply induction on |.M|. In the case M. = 0, the required for- 
mulas immediately follow from the definition. Therefore we assume that 
A4 7^ and put m := rn.rn.Ai. 

Case k ^ I < m — 1. Applying the inductive hypothesis, we obtain 

/WjU^)) = (-^>^l {? - 1} ^\ {m} (^^-i^) 
+^ >i |£ m (J)|^- 1 }4^(£ m (I),/: m (J) i _ 1 ^)x 
xT^ MX{m} (K m (I),K m (J)))/(C(k,m)-u m + u k ) = 

^iMXin^^^JM))) /(C(k, m) - U m + U k ) = 

Case I = m — 1. Similarly to the previous case, applying the inductive 
hypothesis, we obtain 

pf\Tj£j M (I , J)) = {^-^m-i>i\J\ {m ' 2 ^Tj: d ^ M ^ {m ^{I,J m ^2^m-i) 
+5 m - l>t \C m (J)\^ m -^sfl(C m {I),C m (J) m . 2 ^ m ^)x 

xT ^j,M\{rn}( nm ^ n rn(J) U (m - 1))) /(C(k, m) - U m + U k ) = 
— dm-l>i\J\^ m 2 ^ (T k ^J M \{ m y(I, J m -2t-i-m-l) — 

^m(£ m (-0> ^m(^m-2^m-l))^ d ] ^\{ m }(^ m '(^),'^-m(«/m-2^m-l)))/ 

(C(k,m) -u m + u k ) = -5 m _i>i|J^ m - 2 >7^ M (/, J m _ 2 _ w »_i). 
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Case m ^ I. Applying the inductive hypothesis, we obtain 

^P&mV'V) = (-I^I {w} C\ W (Mi-m) 
Mn m (j)\^sfl(c m {i),c m (j))T^ 

(C(k,m)-u m + u k ) = -\J\ {l - 1} (Tff M \ {m} (I,Ji^i) 
-^(^ m W.A B W-i^))^.^ N{m} (^(i),7Z m (J,_ lMl )))/ 
(C(k,m) -u rn + u k ) = -\J\ {l - 1} T$ M {I, J^). 

It remains to prove the last formula. Suppose that i < k. Then applying 
the inductive hypothesis, we obtain 

-<sW(£-(J),£ m (J) U (* - l))Ti% X{m} (K™(I),K m (J)))/ 
(C(k, m)-u m + u k ) = (r^ Um} (7, JU (k - 1)) 

-5^(/; m (/),/; m (ju (fc - i)))r^ ) i _ MUm} (^(/),7e m (ju (fc - 1})))/ 

(C(fc,m) - ^ m + u fc ) = T$ M {I, JU (A; - 1». 

□ 

Lemma 6.6. Let i ^ k < j ^ n and k ^ I < j -1. Then pj 2) (7^jJ ^ (I, J))= 
Oj/I + I^M and 

pf 2) (^(^-0) = -^i^n(,., + i)(^ +1 ( / ) U (0>A + i(J))x 
ifl + leM. Ifi<k then 

= T fe %(/,JU(A ; -l))(n fe -d+|/|i fe - 1 }). 

Finally, we have 

pf-^rijM 1 ' J )) = r £vw( 7 u 0" - l>, ^) J") + «* - d + |/|)+ 

?(^ ( £m(^ ^)) = 3$m(j u o- - 1)> -o- 

Proof. We apply induction on We restrict ourselves to the case 1 + 1 G 

M, since otherwise both p| 2,i ' ) and pf' ,R ^ take all summands of T^ M (I, J) 
to zero. 

In the case M. = 0, the required formulas follow directly from the defini- 
tion. Therefore, we assume M. / and put m := minA4. 



RECTANGULAR LOW LEVEL CASE OF MODULAR BRANCHING PROBLEM 35 

Suppose that i < k. Then applying the inductive hypothesis, we obtain 



^ k %\ {m} (^^( k -mn k -d+\i\^y 



Sl d) (£™(l),£ m (j) U (k - l»(« fc - d+ |£ m (/)|i fe - 1 })x 



k,m 

xT W 



K-^ir i} )(^ MW (^u(fc-i))- 



Sl d) m (£-(/), £ m (JU (fc -l)))x 



xT 



(d) 



iA1Um} (^ m (/), ^ m ( J U <fc - 1»)J /(C(fc, m) - u m + u k 



(u k -d+\I\i^})T^ M (I,JU(k-l)). 

Similar (but simpler) argument gives the formula for Pk^\ (rff) M (I , J)). 
Case I = m — 1. By the inductive hypothesis, we have 

(2,L) (q -(d) (T n ,_ S^CC"(J)U(m-l),£ m (.7)) (C(fc,m)+u fc -<tf |£ m (f)|) 
\ I k,j,M [ - ' >' C(k,m)-u m +u k X 

(2,R) (q -(d) (T m _ si d) m (C™(I)U{m-l),£ m (J)) {u m -d+\1l™{I)\^-V) 
Pi y J k,j,M^ > "~ C(k,m)-u m +u k X 

It remains to notice that \£ m (I)\ = |7fc m (J)|{ m_1 >. 
C 

<S^(£ m (I), £ m ( J)) identically, the inductive hypothesis yields 



Case m ^ / and / + 1 G At. Since in this case ' and p\ ' act on 



(PvISmV'J)) = (-^i, (MW) n( fe ., + i)(£ m 0O u (0, A + i(J))x 

x^t^na+i.,)^ 1 ^'^ 1 ^) U (0) +<i(^(/),£ ? n(J))x 
x^2 + iJMnrm..i + «(^ ,+1 (^ m W) U (0, C l+1 (TZ m (J)))x 



xT, id) 



m,H■l,Mn(m..H■l) , 

j + i^n(i + i..i)( W,+1 (^ m W),^+i(72m(^)) U (/}))/ 
(C(k,m) -u m + u k ) = 

(-^i,A,n( m ., + i)^ +1 (/)U(/),£ m (J))+ 

x^ + i^n(m. J+ i)( wm ( £,+1 W U W).W + iW))) >< 
x2 Wj^n(l + i..i)( R,+1 W.%W u (0)/(C(fc,m) - u m + u fc ) 
-^m,A4n( fc ..m)(^ +1 (/) U <l>,£, + i(J))x 
4?l J ,A 1 n( i+ i.,)^ +1 ( / )^Hi(^)U(0). 



36 VLADIMIR SHCHIGOLEV 

It remains to prove the last two formulas. Applying the inductive hy- 
pothesis, we obtain 

pf-i\^S,M^J)) = { T S,M\ { m } (^(j ~ !>> J ) (C(k,j)+ Uk -d+\I\) + 
^2q£M\{m} ^,i(A4\{m})n(fc..g)(^ 9 (^)' A(-0) X 

-<s^(£-(/), J c m (j))r^. A1Um} (^(7) u (j - i),n m (j))x 

x(C(m,j)+ Um -d+\TZ m (I)\)-Si d) m (C m (I),C m (J))x 

x 'T, q eM\ { m } ^%n( m .. g) (^(^ m W)' £ q (Km(J)))x 
x^Mn^iK^il)) U (j - l),TZ q (n m (J)))])/(C(k,m)-u m +u k ) = 



T, 



U\ W (^ m (/U(j-l)),^ m (J)))x 



x(C(k,j)+u k -d+\I\)/(C(k,m)-u m + u k ) + 

J)) -<S^(£-(^(/)),£ m (^( J))) x 



XT' 



A4n (m .. 9 )(^ m (^( / ))^-(A(«/))))x 



u 0' - 1)^) (C(k,j) + u k -d+ \i\)+ 

(a(i),c q (j))T { 2 Mn(q ^(nv) u (j - i),n q (j)). 

(2 R) (d) — 

Similar (but simpler) argument gives the formula for p)^{T^ M {I, J)). □ 

Lemma 6.7. Let i ^ k < j ^ n and max{/c — 1, i} ^ I < j — 1. Tafce an 
integer m' £ (M. U{k})D(— oo.i + 1], which we call the origin in this lemma. 
If m' = k then 

pf\^ d l M (^ J)) = wi. z+i)+u fe -d+|/|(— ••'])+ 

J+i]n,M T k (J Mn ( k ^(£ q (Ii + i^i),£ q (J))x 

If m' > k then 

Mi\ {l+1} T k % X{m , } (i l+1 ^J)+ 

|/|{H-1} ^q 6 ( m '.J+l]n>l T k d q]Mn(k..q)(£ q ( I l+^)i£<l( J )) X 



RECTANGULAR LOW LEVEL CASE OF MODULAR BRANCHING PROBLEM 37 

Finally, we have pfl^T^l , J)) = T$ M (I U (j - 1), J). 

Proof. We apply induction on \M\. In the case = 0, the required for- 
mulas follow immediately from the definition. Now let Ad ^ 0. We put 
m := min A4. 

Case I < m — 1. We have m! = k. Applying the inductive hypothesis 
with origin at k, we obtain 

x (C(k, l + l) + u k -d+ | 

x(C(k,l + l) + u k -d+\£ m (I)\(-™- l ^/(C(k,m)- Um + u k ) = 
l^l 0+1} ( r ^\{ m }( 7 '+^' J )- 

^ m (£ m (/ i+lH 0A(^)^ w (7l m (/« + w),^(J)))>< 
x (C(fc, l + l) + u k -d+ \I\(-°°^)/(C{k, m)-u m + u k ) = 

\I\ {l+1} T k {d l M {Ii + ^u J)(C(k, l + l)+ Uk -d+ |/|(— -I). 

Case I = m — 1 and m' = m. Applying the inductive hypothesis with 
origins at k and m, we obtain 

pI 3) (^^ j )) = 

(l^^^MH^™- i,J)(C(fc,m)+u, -d+ |/|(— ••—!])- 

u <m - l),£ m (J))x 

X j,A4\{m} )fra->m-l , («/) ) X 

x(u m -d+\R m (I)\^-- m -^))/(C{k,m) -u m + u k ) = 

\ I \ {m} ( r S,M\{rn}^rn-l,J)(C(k,m)+U k - d + |J|(-°° *])- 
^fc,m(^ m (^ mi ^ m -- 1 )' ^ m (^))^rnJ,A4\{m} ("^"(^mi-mi-l)) 'R-m(J)) x 

x(u m -d+|/|(- 0O "' n - 1 ]))/(C(A;,m) -u m + u fc ) = 
x( Um - d + |/|(-°°- m ^ 1 ])/(C(A:,m) - u m + u k ) = 

\ I \^' 1 'kJ,M\{rn}^ Im ^ m - 1 ' J ) + 

\I\ {m} T$,M(Irn~m -1, J)K> " d + |/|(—— H). 

In this case, the summation is empty. 

Case I = m — 1 and m! = k. Beginning as in the previous case, we obtain 
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x (C(k, m)+u k -d+ |/|(-°°- m - 1 ])/(C(A;, m) - u m + u fc ) + 
|/|{W5^(/:m(/ mMm _ 1 ) ; £ m (j))7^.^^ m} (7^m(/ m ^ m _ 1 ) ) 7^ m (j)) = 

l J l {m}r S^( J ™~™-i> J)(C(k,m) +u k -d+ |/|(-oo..m-i] )+ 

|/| {m} ^ (£^„^ 

The last summand corresponds to q = m, the only possible value of the 
summation parameter. 

Case I ^ m and m' = m. Applying the inductive hypothesis with origins 
at k and m, we obtain 

l 1 l {,+1} &w( Ji + w ' J )^' l+1 )^ ~ d+ 



7; 



(d) 



q€(k..l+l]n(M\{m}) 1 k,q,(M\{m})n(k..q) 

(d) 



A&(I l+ M),£ q (J))x 



q,j,Mn(q..j) 



x|^(/)|{'+ 1 }rW._ MUm} (^(7), +1 ^,^ m (j))x 

x (C(m, I + 1) + u m - d + |^ m (/)|(-°°- J ])- 
-|^(/)|^ 1 >[E ?6(m ., + i]nA.<i(^W,>C m (J))x 



xT 



(«0 



m,g,.A/fn(m..q) 



(/:«(^ m (/) i+w ),/: (? (^ m (j)))x 



xT 



(d) 



w -,A<n(,..i)(^(^ m ( 7 ))'+i-i^(^( J ))) )/(C(fc,m) -t^ + ttfc) = 



(d) 



fcj',.M\{m} 



(7,+!^,, J) - £ m (J))x 



x ^\ W r(^M)^ ra (j))jx 

x (C*(m, Z + 1) + u m - d + |/|(-°°- J ])/(C(A:, m) - u m + u fc )+ 
<i(^(^(/ m ^)),£ m (£ ? (J)))x 



xT 



(d) 



^n(,,)r( £ '(Wi))A(^(J))))x 
>< T ifA4n( (? .. J )^( / )'+ 1 -''^( J ))]/( C '( A; ' m ) -«m + «*) = 

l^| {Z+1} ' 7 '5A4\ {m} ( / '+^' J ) + 

|/| {m} r fc %(W ; , J)(C(m,/ + 1) + u m - d+ |J|(— 
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J2 qe (m J+l]nM T k d q\MC\{k..q)(£ q ( I l+^)^<l( J )) X 

Case I ^ m and m' = k. Handling the sums in the square brackets as in 
the previous case, we obtain 

P? ] (r k %(I,J)) = 

l 7 l { ' +1} ( r S^\w^+^' J ^ C ^ l+l ^ +Uk ~ d + l^ 00 "' 1 )- 

x (C(m, l + l)+u m -d + |/|(-°°--'])) /(C(k, m)-u m + u k ) + 
Z q e { m..l + i]nM ^n( fc ..,)( £, (W/), A( * 

x (C(fc, Z+l)+u fc - d + |/|(-°°- J ])/(C(A;, m) - u m + u fc ) + 
|/|^+ 1 >5^(£ m (/ /+1 ^0,^(J))^^^ Um} (^(/), +1 ^^^ m (J))+ 

|J|{'+ 1 > Y,qe(m..l+l]nM Tk d q ] Mn ( k .. q )(£ q (Il+l^l),£q{J))x 

x ^n(,.# , Wi+iHi^(J)) = 

J)(C(M + 1) +u fc - d + |J|(— "'])+ 

I Z ge (k. .l+i\nM Tk%Mc\(k..q) ^ i^+^l )^g( J )) x 

Case / ^ m and m' > m. Applying the inductive hypothesis with origin 
at m' , we obtain 

/ l {m}T ifA.\ M (W^)(C , K,/+l)+n m ,-a'+|/|(---']) 

+ l / l {,+1} ^\ { m,m'}( / I+l-I.-0 + 

+ J2qe(m'..l+l]nM' T 'k,q,(M\{m})n(k ..,,)(^^+m))^(^)) X 

xT ^n (? .. J . ) (^(/)w^,^(^))]-|^ m WI { ' +1} 41(^ m W,>C m (J))x 

-|^(/)|^}5W(£-(7),£ m (J))r^ M{mm , } ^ 
|^(/)|{'+i}4^(£-(/),£ m (J))x 

x E, 6(m ^i + i]nM^^n(m.. s) (^(^ m W^).A(^(- 7 ))) x 
x^n(,..i)(^(^ m ( / ))'+i-i^(^( J )))])/( C '( fc » m ) + = 
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l / l { ' +1}T 5A4( / m^^)(C(m',/+l)+u m ,-d+|/|(- 0O --'])+ 

+\i\ {w} ^S M \ {mf} i.ii+i^,j)+ 

+ X] 9 e(m^J+l]nA4(^,^A4n(m.. g )(' C<^ (^+l^)'A( J ))- 

si%(£ m (C<i(I l+1 ^ l )),£ m (C q (J)))x 
xT I d Un( m .. g )(^ m (^(W/)),^n(A(^))))x 

Xr &n(,j)(^( / )i+W'^( J ))]/( C ( fc ' m ) -U m + U k ) = 

|J"| {m} r£; M (/ m ^,J)(C(m'J+l)+n m ,-d+|/|(— ••'])+ 
+l^l {m}r SM\ {m ' } (^i-^)+ 

+ E 9e ( m ,.j +1]nA4 T fc ( ^_ Mn(/ ., q) (C q (Ii +ll ^i),Cq(J))x 

To prove the last formula, we apply the inductive hypothesis and obtain 



j!aW J U (j - 1), J) - 5^(^(1 U (j - 1)), An(L))x 



'A' 



( 

^%au(j-i),J). 

□ 

It would be natural to ask whether the operators pf \ pf\ take 
integral polynomials to integral. Although the answer is actually affirmative, 
we prefer to work only with integral polynomials of the following special 
form. 

Definition 6.8. A polynomial of of the form 

where i=mo<mi< ■ ■ ■ <mk<mk+i=n, M.\ C (mo. .mi), M.2 C (m\. .m^), 
. . . ,Mk+i C (mfc..mfc + i), / £ U°, Jo = ((« — l) d ) and conditions |(M1)| - 
|(M3)| hold for the sequence of multisets I\, . . ., Ik+i, Ji, ■ ■ ■, Jk and the set 
{mi, . . -,mk} is called a T -monomial and Xi „, KA (Ik+i, Jk) is called 
its tail. 

Clearly, (JO} has weight A^ ){mii .„, mk }(h, ■ ■ ■ ' J l,---,Jk)- 
Corollary 6.9. Any T -monomial is an integral polynomial. Each operator 
Pi > Pl^ > P?^ takes a T -monomial of weight X to a sum of T -monomials of 
weight A + ci\ . 



Proof. The first statement follows from Lemma I6.^(iii) representation given 



in the proof of Lemma I6.^(i)| and the remark of Section 16.21 The second 
statement follows from Lemmas I6.5H6.7I and Lemma 13.21 □ 
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6.5. Raising coefficients. Based on Lemma |4.1| we define the ring homo 
morphism cf ^ : — > U° that acts identically on U° and acts on the formal 
operator S^ m , (I, J) as follows: 



11 I* 7 ' ■(C(m,t)+u m -\J\l t -+°°y)-(C(m,t)+u m -\J\l t ~ 1 -+^)-q t ) V ' 

t=m+l 

where x = (<ft+i, . . . , <z n ) is a sequence of nonnegative integers. 

It follows from (|6.ip that no cutting operator takes the denominator of 

cf ^( 5 m,m'( J ' J )) to zero - 

Lemma 6.10. Lei P be a regular polynomial of F^ and i ^ m < m' < n. 

Then we have 

(o"m,m' ° ci* o cr mjm /) LP • (C(m, m') - u m > + u m fj 

= (o- m ,m' ° cf*) LP • (C(m, m') - u m i + ii m )) . 

Proof. Since <7 m ,m/ and cf* are Z-linear, it suffices to assume that P is 
a regular monomial. Let P = Q ■ f/g, where f,g 6 g ^ and condi- 
tions (F3) - (F5) hold for Q and g. If g is not divisible by C(m, m') — u m i+u m , 



then <7 m)?n ' is applicable to both P and cf^(P). Hence by virtue of the re- 
mark of Section 16.21 it follows that both sides of the required equality equal 
zero. 

Now consider the case where g is divisible by C(m,m') — u m i + u m and 
denote by g their quotient. By conditions |(F4) and (F5)[ we obtain Q = 
QoS^ m „{I , J), where m' ^ m" and Qo is a product of formal operators of 

the form S^ M (I, J) with [m..m') n [m..m") = 0. Thus we get that o m ^ m i 
acts identically on Qo- Moreover, if m! = m" , then a mm i acts identically 
on the whole of Q and the required equality immediately follows, using the 
fact that o m ^ m i is an idempotent operator on U°. 
Therefore, we consider the case m! < m" . We have 

(o- m ,m' ° cf>f o <j m<m i) (p ■ (C(m, m!) - u m i + u m )\ 

= {o- m ,m> °cf^)((5o • ^ m ' m ',^l ' 0-m,m'( 5 mm"(^' 

\ 777 , TfL \J / / 

= %y (cf^(Qo) • U ^J§) ■ (Cfx O ffm,m')(^ m //(/, «/))) 

771 , 777. V f / 



(^m,m' 

o cf (p • (C(m, m') - n m / + u m )J 

= 0"m,m' (cf^(Qo) • I • cf^S^,,^, J))J 

= (<r m , m < o cQ(Qo) • • O mX o ef„)(<S^ m „(I, J)). 
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It remains to prove that 

(<T m ,m' ° cf* o <r m , m >){S^ m „{I, J)) = (<7 m>m , o cf„)(«S^„(/, J)). (6.5) 
We put 



m' — 1 / 

<t> — TT [|/|{*- 2 }i (gMi!!i)^ 

11 l<V| •(C(m,t)+rt m -|J|[*--+°°))-(C(m,t)H 



-oo..t-l] 



t=rre+l 

X 



(C(m,t)+« m -|J|[*-+°°))-(C(m,t)+« m -|J|[ t - 1 -+°°)-g t )' 



/ |J| {t_1} \^|Ji(---'-']-|j|['-'"+«)\ (ff _ f,^) 

^|7|{*-i}-|J|{*-2}+ ? J^ qt )( H t-l ti t )-\ 



TT ( I 7|{*-2}, (C(m',i)+n m ,) j+1 -l J ' ( °°" t 11 

11 l J l •(C(m',t)+« m ,-|J|[ t -+°°))-(C(m',t)+« m ,-|J|['-l-+°°)-gi) 



rn 

X 

t=m'+l 

X 



We clearly have 



, , d+i-i/it- 00 "™'- 1 ] 

I T|{m'-2}, u m , 

' ' • (Urn>-\J\ [m '-- + ~ ) ><U m >-\J\ [m '- 1 -- + °° ) -<lrn>) 

|/|{m'-l} \ / d _| 7 |(-oo..m'-l]_| i/ |[m'-l..+oo)\ 

|7|{m'-l}_|J|{m'-2} +gm/ ^ 9m , ^ 

XCJ rnim /(iJ m /_i - # m /)— • 

We can rewrite 
m'-l / 

*:= II |/: m '(J)| {t - 2} !x 



(6.6) 



t=m+l 



(C(m,t)+M m -|£ m ,(J)|[*-+-))-(C(m,t)+« m -|£ m ,(J)|['-i-+-)-g t ) 



X 



|£m'0O|{t-l} 
|£-' (J) | {t -l}_| £m/(J) | {t -2} +gt 



x^-tf^ H i\K m ,(J)\^h 

/ t=m'+l V 
(C*(m',*)+« m /)- 



(C(m',t)+« m ,-|W m ,(J)|[«--+'»))-(C(m',t)+ti m ,-|W m ,(J)|[*-i--+«»)-g t ) 



|^m' (/) |{t-l} \ / d _| W m'( J) |(-oo..t-l]_| 7Jm ,( t7) |[t-l..+oo)- 

i7^'( 7 )|{*-i}_|^ m/ (j)|{*-2} +gt n qt 



)( 

X(^_!-^)^) , 
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whence we obtain 
(cf^ o er miTO /)(«S^ m „(I, J)) = 

d«(s^ m/ (£ m '(i),£ m >(J))) ■ d^l m// (n m '(i),n m/ (j))) 

9 Um |£ m /(J)|t-+») Mm> |^(J)|t"''--+°°) | £m , ( J)|{m'-2} !x 
(C( m , m ')+n m )j+ 1 -l^ , (l)|(~ , - 1 i 



(C(m,m')+Wm)-(C(m,m')+ ! im-|£ m '(J)]I m '- 1 -+ 00 '-9 m /) 

| £ m' (/) |{m'-i} Vd-|£ m '(/)|(-°°- m '- 1 l-|£ m ,(J)|[ m '- 1 -+°°) 

| £m ' (/) | {m '-l}_| £m , (J) | {m '-2} +gm , ^ , m , 

Noting that u m ^ m i acts identically on <I>, we obtain 

/ r CT T\\ ^ |J|[m.. + oo) |j|[m'.. + oo) 

(ffm,m' ° ct„ o 0- mjm /)(5^; m „(/, J)) = $> U m J-J U m /-U ; 

d+l- 



x | 7|{m / -2|| V 

11 '« m '-(« m '-|-/| lm '- 1 " + ° o) -9 m ') 



j I ( — OO. .m' — 1] 

X 



X 



|/|{m'-l} \ /^_|j|(-oo..m.'-l]_|j|[m'-l..+oo) 

| 7 |{m'-l}_|j|{m'-2} +?w I ^ g m , 



Comparing this with (|6.6I) gives (j6.5j) . □ 
Note that applying ci H to a formal operator does not always give an 
element of U°. However, this is true in all cases of interest. 

Lemma 6.11. If we apply cf^ to a formal operator of weight ^ 0, then we 
obtain an element of 11°. 

Proof. Let S^j ,{1 , J) be a formal operator of weight ^ 0. For any t = 
m+1, . . . , m 1 , we have d+ 1 - |X[ C" 00 --*- 1 ] > d- _ | j|[*-l-+oo) ^ Q 
Therefore cf x{S^ m , (I, J)) = except the case where qt ^ al — |/|(~ 00 - i_1 ] — 
|j|[*-i..+oo) for all t = m + 1,... , m', in which the denominator divides the 
numerator in each fraction of (|6.4[) and thus cf x(«5^ m , (I, J)) G U°. □ 

Corollary 6.12. Let P € F^z! be an integral polynomial of weight ^ 0. 
Then cf x (P) eU°. 

Proof. Consider the representation cf^P) = A/B, where A, B G U°, B ^ 
and the fraction A/B is irreducible. By property (F4)| of the definition of 



a regular polynomial, all formal operators occurring in P have weights ^ 0. 
Therefore, by Lemma [6.111 and property |(F5)| we obtain that B is a product 
of polynomials of the form C(m,m') — u m ' + u m , where i ^ m < m' < n, 
each occurring in degree at most one. 

Lemma 16. 101 together with the definition of an integral polynomial yields 

a m , m '{j}{C{rn,m') - u m > + u m )j =(%,m' ° cf^) (P-(C(m, m') -u m > +u m )) 
= Om,m' ° cf ^ o a m!m ')(P ■ (C(m, m') - u m > + u m )) = 0. 
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Suppose that B is divisible by C(m, m') — u m i + u m , where i ^ m < m! < 
n, and denote their quotient by B. We have <J m)m i(B) ^ and 

= a m)m '(A/B) = a mtm i(A)/a mtm /(B). 

Hence <J m ,m' (A) = 0, whence by Lemma 16. H we obtain that A is divisible 
by C(m,m') — u m i + u m . This contradicts the irreducibility of A/B. Thus 
we have proved that B = 1. □ 
In the sequel, we shall use the following notation 



j 



M (I j\ ._ TT \J\{t-*h( 

t kj,x\ 1 > J )— 11 \ J \ ^|J|{t-l}_|J|{t-2} +9t 
t=k+l 



X 



x(^l ( -^ 1 H J l [t - 1 " + - ) )(^ t _x-^. 
For k < m < j, we have 

<U J > J ) = <i,^ m ( 7 )' £ ™( J )) cf^,^ m (/),^ m (J)). (6.7) 
6.6. Back to the hyperalgebra. A monomial of of the form 

S#U(/i, Jo)S^, m2 (l2, Ji) ■ ■ • S%l mk+i (I k+1 , J k )f/g, (6.8) 

where i = mo < mi < ••• < < m^+i = n, /, 5 € 9 / 0, 



J = ((i-\) d ^ an d conditions ["(Ml)H(M3)| hold for the sequence of multisets 



Ii, . . . , Ik+i, Ji, . . . , Jfc and the set {mi, . . ., nik} is called full (cf. Defini- 
tion [6T8]). A polynomial of i* 1 -^ is called full if it is a sum of full monomials. 
It follows from Corollary 16.91 that any T-monomial is an integral full poly- 
nomial. 

We define the function ev with values in U~'° as follows. Let the value of 
ev on (]6.8p be 

l {m 1 ,..,m t (^ 1 ' • • • >Ik+u Ji,---, Jk)f/g- We extend ev to 
an arbitrary full polynomial by linearity. 

Lemma 6.13. Let P be a regular full polynomial and i ^ m < ml < n. 
Then we have 

(a m!rn 'oev)(P-(C(m,m')-u rn/ +u m )) = (evoa mtrnl )(P-(C(m,m')-u m >+u m )). 



Proof. It suffices to consider the case where P equals (|6.8f) and (F3) - (F5) 

hold for Q := 5m], mi (h, Jo)Sm\,m 2 (h, J\)--- <Sm k ,m k+1 (4+1, Jk) and g. It 
was proved in Section [3] that ev(Qf) € IA~^. Therefore, if g is not divisible 
by C(m, m')—u m >+u m , then <7 m ,m' is applicable to both P and ev(P). Hence 
both sides of the required equality equal zero by the remark of Section 16.21 
Now suppose that g is divisible by C(m, m!) — u m > + u m . Condition |(F5)| 
then yields that there exists some r = 0, . . . , k such that m = m r and 
m' ^ m r+ i. Now it remains to apply Lemma 16.21 □ 

Corollary 6.14. Let P be an integral full polynomial. Then ev(P) € LC'°. 

Proof. This result can be proved similarly to Corollary 16.121 Indeed, con- 
sider a representation ev(P) = A ■ B" 1 , where A G U~ ,Q , B £ U°, B ^ and 
B has minimal possible degree. Since applying ev to a full monomial being 
a product of formal operators gives an element of U~ ,Q , condition | (F5) | of the 
definition of a regular polynomial yields that B is a product of polynomials 
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of the form C(m, m') — u m i + u m , where i ^ m < m! < re, each occurring in 
degree at most one. 

Suppose that B is divisible by some polynomial C(m,m') — u m / + u m , 
where i ^ m < m! < n. Let B denote their quotient. Lemma 16.131 yields 

= (ev o cr m>m ')(P • (C(m,m') - u m > + u m )) 
= o- m>m >(ev(P) ■ (C(m,m') - u m > + u m )) 

= <7rn,m>{A • B^ 1 ) = <7 rrijrn i(A) ■ <7 mjm ' (P) _1 . 

Hence a mjm i(A) = 0, whence by Lemma EH we obtain A = A' {C{m,m!) — 
u m i +u m ) for some A' £ U~'°. This fact contradicts the minimality of degree 
of B. Therefore B = 1. □ 
To formulate the next result, we introduce the function cf on any full 
polynomial P as follows: 

cf (P) := y^jcf >i{P) | x a sequence of nonnegative integers of length n — i}. 
Since P is full, only finitely many cfj^P) are nonzero. 

Lemma 6.15. Let P be a full polynomial of weight — a\a\ — ■ ■ ■ — a n _ia n _i, 
where a\, . . . , a n _i ^ 0. Then 

a n _!! • Pj ai) • • • Pi a _" 1 - l) ev(P) = cf(P) (mod J+). 
Proof. The result follows directly from Lemma 14.11 □ 
We put := pf' + + p[ 3 \ The next lemma explains the role of this 
operator. 

Lemma 6.16. Let P be a full polynomial and i ^ I < n. Then we have 

Ei ev(P) = (ev o Pl ){P) (mod f+) 
Proof. The result follows directly from (|3.10p - (|3.12p and the definition of 

To prove Lemmas 16.151 and I6.16[ it is useful to notice that ev is linear in 
the following sense: ev(P + P') = ev(P) + ev(P') and ev(Px) = ev(P)x, 
where P and P' are full polynomials and x £ U°. 

Lemma 6.17. Let L be a multiset with entries in [i..n), J = ((i - if) and 
A4 C (i..n). Suppose that A4 ^ and denote m := min.M. Then we have 

^(^(PJo)) = 



ev(r (d) 



i,n,jvl\{m 



} (I,J )) -°"i,m(ev(7:^ M{m} (/, J )))Vc(«,m) -u m ) 1 . 



Proof. Applying Lemma [6.131 with P = T^ m (I,Jq), Definition 16.31 and 
Lemma 16.41 iii) we obtain 



Km°ev)(^ V{ro} (I, J )-sW(£ m (I), Jo) ^ M{m} (^(/),0)) = 0. 
Thus Lemma 16.21 (the second case) implies 
ev(sS(/-(/),J )^ 

Now it suffices to apply ev to T^n M (L, Jo), use Definition [63] and the remark 
on linearity preceding this lemma. □ 
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6.7. Polynomials cf ^(T^ d ) M (I, J)). Throughout this section, we fix a se- 
quence x = (f/i+i, . . . , q n ) of nonnegative integers. For any integers k and j 
such that i — 1 ^ k < j ^ n and a multiset J with integer entries, we put 

A^-(J) := {(t, h) G Z 2 | k < t ^ j & | J|[*"+°°) <; h < | J| [*-!••+«>) + ft J , 

where % = 0. We obviously have 

Sg(^(/))=sg(/), A^(£ J (J))=A-.(J)u({i}x[0..|J|b--+-))) J 

S g(^(/))=sg(/), A*.(7^(J))=A*.(J). 

Proposition 6.18. Lei 1 ^ m\ < ■ ■ ■ < mi ^ N be integers and fx, . . . , /; 

6e polynomials of Z[xi, . . . ,xn] having the form f s = x mg — g s , where g s is 
an integral linear combination of the unit and of the variables xt for t > m s . 
Let X be the ideal of Z[x\, . . . , xn] generated by fx, . . . , //. Then 

(i) I is a prime ideal; 

(ii) an integral linear combination of the unit and the variables x% be- 
longs to X if and only if it is an integral linear combination of 

hi ■ ■ ■ > //• 

Proof. Applying nondegenerate linear transformations, we can without loss 
of generality assume that the polynomials g\ , . . . , gi do not depend on the 
variables Xm x , . . . , . 

Let if be the ring endomorphism of Z[x\, . . . , xjy] that takes x ms to g s for 
any s = 1, . . . , I and acts on the remaining variables identically. We claim 
X = ker (p. Indeed (p(f s ) = (p(x mg ) — g s = 0. Hence (f(X) = and X C ker <p. 
Now suppose on the contrary that (p(f) = for some / G Z[xi, . . . ,xn]. 
Consider an arbitrary monomial v of / represented in the form 

v = x mi ' ' ' x rni ' M ; 

where u is a product of powers of the variables Xt with t G {1, . . . , N} \ 
{mi and an integer. We have (f(v) = g^ 1 ■ ■ ■ gf l ■ u. Considering the 

representation 

v = (fi+9i) ai •••(// + 9i) ai -u 
and applying to the first I factors the binomial theorem, we get v = f(v) 
mod X. This extends to / = <p(f) = mod I. Hence / G X as required. 



(i) Let /<? G T. Then we have = <p(fg) = <p(f)<p(g)- Hence <p(f) = 



or ip(g) = 0, since Z[xi, . . . , scjy] does not contain zero divisors. Therefore 
f (£X or g ex. 



(ii) Let / G X be an integral linear combination of the unit and the 



variables x%. We write it as 

/ = b\x mx H V bix m - w, (6.10) 

where w is an integral linear combination of the unit and the variables xt 
with t G {1, . . . , iV} \ {mi, . . . , mi}. We have 

= <p(f) = hgi H h fyffi - «>■ 

Hence u; = b±gi + • • • + &/<?/. Substituting this back to (|6.10j) . we obtain 
/ = bi(x mi — <7i) H h 6/(x m , - flf|) = 61/1 H h fy/j. 

□ 
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Lemma 6.19. Let k,j,I,J and M. be as in Definition \6.3l Suppose that 
A^.(J) C S^.(J) and there is an injection i : M -> S^-(J) \ A^.(J) 

suc/i i/iai t for any t £ M and ti(t) = i implies i2{t) < \J\^ t " +oc \ 

where t(t) = tv(t)). Then modulo the ideal 2 of U° generated by 

C(t, L\(t)) + ut — t2(i) for t G we /taue 

cf4^(I ) J))=^^^cfg > (/ ) J)x 

x n{c(^> t) + u k -h\ (t, h) E (sjg(J) \ A^-(J)) \ Im t }. 

Proof. Notice that I has the form described in Proposition [6J2J To see this, 
one can put xi := u i+ \, . . . , x n _i_; := u n -i, x n -i := Hi,... ,x 2n -i-i ■= H n 
and take mi := 1, . . . , m n _i_j := n — 1 — i (I = n — 1 — i, N = 2n — i — 1). 

We note that the condition A^.(J) C S^j(J) is equivalent to qt ^ 
d - |j|(-°o..t-i] _ |j|[t-i..+oo) for any t = + 1, . . . , j. Thus T$ M {I, J) 
is an integral element of weight 0, whence by Corollary 16.121 we get 
c£ x (T£j M (I, J)) € W°. Therefore, both sides of the equivalence we must 
prove belong to U°. 

We apply induction on \M\. In the case A4 = 0, the result follows 
from (|6.4|) . Now suppose that M / 0. We put m := min.M and e := 
C(k,m) — Um + Uk- We shall use the equality C(k,t) + u k = e + C(m,t) + u m . 
Consider the restriction ip := i\M\{m\- ^ follows from (|6.9p that ip is an 

injection from M \ {m} to £^.(ft m (I)) \ ^{K m {J)). 

Applying the inductive hypothesis and (|6.7p . we obtain modulo X the 
equivalences 

= • c Ur k %(i, J)) = cf.(r^ X{m} (i, j)) 
-tf*(s&(/m £»(■/))) cf.(rW MV{m} (^(i),^ m (j))) = 

x H{c(k, t)+u k -h\ (t, h) E (£<$(/) \ A^-(J)) \ Im ^} 

x n{c(^, *) + u k - h | (t, h) G \ A£ m (£ m ( J))} x 

Xu J^'^ Cfg 3> (^(l), IZm(J)) X 

x n{c(^> + «m - /» I (t, h) e {^{TZ m (I)) \ A^(n m (J))) \ Imip] 
= u fc ^ cf^J.J/, J)x 

x t)+u k -h \ (t, h) € xgj£ m (/)) \ A£ m (£ m ( J))} x 




x + C(m, t) + « m - h | (t, /i) G (S^.(7) \ A^.(J)) \ Im</>}- 
^M^^) Jj{ C (m, t) + u m - h | (t, fc) € \ A^(J)) \ Im </] 
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Let X denote the polynomial in the square brackets. 

(T) \ A 

m,j V / \ raj 



Case 1: i\{m) > m. We have t(m) G (E^-(I) \ A^j ,-(</)) \ Im?/;, whence 



„ / \ I J\lm.. + oo) 

X = e(e + u m )HJ x 

x + C(m, t) + n m - /i | (i, fc) G (E^. (J) \ A^.(J)) \ Im t } 

+(C(m, + u m - t 2 (m))F. 

In this sum, the last summand belongs to 2 and 

, I rl [m. . + oo) 

y = (e + u m )^ -x 

x H[e + C(m, t) + u m - /» | (t, h) G (E^.(I) \ A^j(J)) \ Im,}- 



| j-|[m.. + oo) 



n{c(m,t)+n m -/ l | (t,h) G (E^-(J) \ A^.(J)) \Irnj 



Substituting X written in this form back, we get 

v (6.11) 
x n{C(^ t)+u k -h \ (t, h) G (E]g(I) \ A^-(J)) \ Im,}) G J. 

Since £ depends on Hf. and no polynomial C(i, t\(t)) + u t — i 2 (i)j where 
t G A4, does, we obtain by Proposition I6.1^ii)| that e ^ 2. Applying 
Proposition I6.1^i)[ we obtain the required equivalence. 

Case 2: ti(m) = m. By hypothesis, t 2 (m) < |j|[ m --+°°). Therefore 

X = e H{e + u m -h\h€ [0..| J|K-+°°)) \ {, 2 (m)}}x 

x + C(m,t) + « m - h | (t, fc) G (S^.(/) \ A^-(J)) \ Im ^} 

+ (u m - L 2 (m))Y. 
In this sum, the last summand belongs to 2 and 

y = ~[[[e + u m -h\h£ [0..| J|K-+-)) \ {, 2 (m)}} x 

x + C(m, t)+u m -h\ (t, h) G (E^.(/) \ A^.(J)) \ Im ^}- 

J]| Um - h \he [o..| j|[--+-)) \ { i2 (m)}} x 

x n{c(m,t) +n m -h\(t,h) e (E<* (I) \ A* J)) \ Im^}. 

Hence (|6.11|) follows similarly to case 1 and the argument of that case com- 
pletes the proof. □ 

Lemma 6.20. Let k,j,I,J and Ai be as in Definition \6.'J[ Suppose addi- 
tionally that T^ d j M (I, J) has weight ^ 0. Then ci^iT^ J)) is divisible 

byd^ JI,J) inU°. 

Proof. Throughout this proof, we should keep in mind Corollary 16.121 

We apply induction on \M.\. Consider the case M. = 0. If qt ^ d — 
| j|(-oo..t-i] _ | j|[t-i..+oo) £ or an y ^ _ k + 1, . . . ,j, then the denominator 
divides the numerator in each fraction of (|6.4p . where m = k and m' = j, 
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and the result follows. On the other hand, if this condition is violated, then 
Now suppose that A4 7^ and let m := min.M. We have 

cf -(^( J . J )) rn) - u m + u k ) = cf.(T^ MUm} (/, J))- ^ 

cf x (5^(£-(J),£ m (J)))cf.(rW. M{m} (^(/),^ m (J))). 

By the inductive hypothesis, d 3t (sj^(C m (I), C m { J))) is divisible by 
<l,.(£"mAn(J)) and c^(T^ Um} (^(I),7e m (J))) is divisible by 
cf^j ^(lZ m (I), lZ m ( J)). Hence their product is divisible by 

rf^l,^(>C m (I),>C m (J)) cf^.^(^ m (/),7e m (J)), which by UEZD equals 
cf^f] ^(1, J). By the inductive hypothesis the last polynomial also divides 
cf x(1~fr d j M \{ m y(I , J)). Therefore the left-hand side of (|6.12p is divisible by 
c ffcj JJ-i J)' Since C(/c, m) — u m + is not a prime factor of cf j^j J), 
we obtain that cf ^(T^ d j M (I, J)) is divisible by cfj^j J) as required. □ 

Remark. Suppose that T^ d j M (I, J) has weight < 0. Then cf^j H (I, J) = 
unless q t ^ d - |T[ C" 00 -*- 1 ]' - | j-|[*-i--+oo) and |j|{*-2} _ <; Qt ^ 

|j|{*-2} for any t = k + 1, . . . , j. In particular, cfj^-^I, J) = unless 
A^.(J) C sgJ(Z), |/| < d and |J| d. 



7. Proof of the main result 

7.1. Operators m(I)- In this section, we fix an algebraically closed 
field K of characteristic p > and denote by fh the sum lj? + • • • + Ik 
with m summands. Using the notation introduced in Section 12.51 we have 
m T = fh for any homomorphism r : Z[ttj+i, . . . , u n -i] — > IT. 

We suppose that d < p. Any rational GL n (i ; C)-module V can be consid- 
ered as a [/^-(n)-module (see [Jj 1.7.11]), since C/^-(n) is naturally isomorphic 
to the algebra of distributions Dist(GL n (i^)) (see [Jj II. 1.12]). 

In particular, in the sense of Section [H a vector v € V has weight A £ Z" 
iff ( H r s )v = ( r s )f for any s = l,...,n and r G Z; a vector d G V is a 

GL n (/A~)-high weight vector (GL n _i(iif)-high weight vector) iff £g v = for 
any r > and s = 1, . . . , n — 1 (resp. s = 1, . . . , n — 2). 

Throughout this section we use the notation Jo := ((i — l) rf ). 

For a subset Mc(i..n)xZ having at most one point in each column and 
a multiset / of length no greater than d with entries in [i..n), we define the 
following element of the hyperalgebra Uk(ti): 

rfUW : =ev(^t l(M) (J,Jo)) r , 
where r is any ring homomorphism from Z[uj+i, . . . , ii n _i] to -fT such that 



T{u t ) = h if (t, /i) € M. 



(7.1) 
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Since ev (^„ 7ri /jvf)(-^' ^o)) depends only on the variables ut with i G 7i"i(M), 

the element ^^AfCO does not depend on the choice of particular r satisfy- 
ing flUD. 



7.2. Subspace V>. For any A G X + (n), we consider the subspace V\ of 
L n (A) spanned by all vectors ev(u) T v^, where u is a T-monomial of weight 

^ 0, which we write as (|6.3p . such that for some subset M of ^£2.-1 nih+i) 
having at most one point in each column there hold 

(a) tti(M) = {m k }UM k +i; 

(b) dist^(x,y) = (mod p) for any x,y G M; 

(c) r («t) =h if G M; 

(d) there exists a strictly increasing injection 99 : M — > Xm],,n(4+i) 
such that dist,\(a;, </?(x)) = (mod p) for any x G M. 

We claim the following properties of the vector ev(u) T v^ just mentioned: 

(i) • • • E^-i ev(u) T vt = 0, where u (and, therefore, ev(u) T ) has 
weight -aiai a n _ia n _i; 

(ii) £^e\{u) T v^ G "V 7 ^ for any I = 1, . . . , n — 2 and r > 0. 



To prove (i) we note that by Corollaries 16.91 and 16.141 we have ev(u) G 
U~'°. Corollary 16.121 implies that cf(u) G U°. Hence it follows from Lem- 

mas[6Ji2andE3]that a n _i!-^j ai) • • • E^^evfu) = cf(u) (mod J+). There- 
fore, since (I + ) T v^ = 0, we obtain 



(7.2) 



We are going to prove that cf(u) r w^ = 0, from which property (i) will 
follow, since a n _i ^ d < p. Choose any sequence x = (qi+i, ■ ■ ■ , q n ) of 
nonnegative integers. Each factor in product (16. 3D is an integral polyno- 
mial of weight ^ 0. Thus if we apply cf ^ to any of these factors, then by 
Corollary 16 . 12 1 we shall get an element of U°. Hence it remains to prove that 

^U +1 fei,4))V = (7-3) 

(recall that cf^ is a ring homomorphism). 

By Lemma 16.201 and the remark following it, (17. 3p is satisfied unless 

|J fc |^- 2 >-|4+i| {t - 1} < % ^l<4l { *- 2} 

for any t = m/% + l,...,n. Now assume that inequalities (17. 4p hold. There- 
fore, A* k<n (J k ) C £^j., n (4 + i), |J fe+ i| ^ d and |J fe | ^ d. 

We shall use, the theory developed in Section 15.41 for a = m^, b = n 
and S = A^ fe _ l n (Jfc). The conditions imposed on S in Section l5~H follow 
directly from (|7.4p . Let X := M U line/?. Note that any point of X not 

belonging to column belongs to T,m\,n(Ik+i) and that distA(x,y) = 
(mod p) for any x, y G X. The last assertion follows from properties |(b)| 



and (d) 
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If X n S contains two points comparable with respect to <, then by 
Lemma [5. 6 1 we have (At — At+i)^ii = (mod p) for some t = rn/., . . . , n — 1. 

Therefore {cim k ,n,H{Ih+li Jk)) T v^ = 0, whence (|7.3|) follows by Lemma f6.2Ul 
Now suppose that X PiS does not contain points comparable with respect 

to <. In this case, we can apply Lemma 15.51 and obtain an injection ips : 

M — » X \ S satisfying conditions described in that lemma. One can easily 

observe the following property of this injection: 

(d') dist A (x, (ps{x)) = (mod p) for any x G M. 
We put i(t) := ips((t, h)) for any t G A4k+i, where (t, h) G M, and apply 



Lemma 16.191 for this injection. Applying property (c) we get 



{C(t, n(t)) +U t - fc 2 (*))X = dist A ((t, h), <p s {(t, h)))v+ = 



by property (d') Here and in what follows i(t) = {ti(t), ^(t)). This implies 
1 T v^ = 0, where X is the ideal of U° generated by all C(t, ii(t)) +ut — L2{t) 
for t G Mk+i- Thus Lemma 16.191 implies 



k 



n{ 



t - m k + \ mk - At + t) - h | (7.5) 

(t,/0 G (Sffi,«(4 +1 )\ A^ fcift (J fc )) \Im t }«+, 

where (mfc, f)) G M. Since ^ f) < p, property (c) implies f) = if = i. 

We put (91,92) := L Ps(( m k> f)))- First consider the case (m&, f)) < (91,92)- 
In that case (91,92) is a point of X not belonging to column mk, which 

implies (91,92) G Sml,u(4+i)' On the other hand, (91,92) ^ S', whence 
(9i)92) ^ ^m k ,n{Jk)- Since </?g is an injection, we have (91,92) ^ Imt = 

p s (M\{(m fc ,f>)}). We have proved that (91, 92)G(ES, n (7 fc+1 )\A^ in (J fe ))\ 
Im t. To prove (I7.3p it remains to notice that 



9i-"^+A mfc -A 9l +f} - 92 = dist A ((m fc , h), (91,92)) = 

by property |(d')| a nd substitute this to (I7,5p . 

By Lemma l5.5| it remains to consider the case where (mfe,f)) = (91,92) 
and this point lies below 5, that is < | J k \^~ +00 \ Hence ftl J *l [mfc " +0, ' ) = 0. 
Substituting this back to (I7.5p . we get (I7.3p . 

Now let us prove property (ii) We can obviously restrict ourselves to 
the case i ^ I. Since a; ^ d < p and S^ ev(u) T vt = if r > ai, it 
suffices to consider the case r = 1. By Corollaries 16.141 and 16.91 we get 
ev('u), (ev o pi)(u) G Vi~ , °. Therefore, it follows from Lemmas 16.161 and 12.31 
that Ei ev(it) = (ev o pi)(u) (mod Since (/ + ) T w^ = 0, we obtain 

Si ev(u) T v+ = {Ei ev(u)) T v+ = (ev o Pl )(u) T v+. (7.6) 

Now let us take c = 1, 2, 3 and use Lemma |6.5[ Lemma 16.61 or Lemma 16.71 
respectively to represent p^ c \u) as a linear combination T-monomials. Let 
w be any of the T-monomials occurring in this linear combination. We 
must prove that ev(w) T v~^ G V\. For this, we clearly can assume that w has 
weight $C 0, since otherwise ev(w) = 0. 

If / < mj; — 1 then the tail of w is the same as tail of u (see the remark 
of Section 16. 3p and thus satisfies conditions (a) - (d) Thus we shall consider 
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the case m k — 1^1 and see for what sets and injections the tail of w satisfies 



conditions (a) - (d) 



Case I: c = 1 . By Lemma 16. 5\ we obtain that the tail of w is either 



x 



(d) 



m k ,nM k +M k + 1 ' ( J k)l-i~l) tfm k ^l or Z^l n>Mk+i (I k+1 , J k U (m k - 1)) if 
I = m k — 1. In both cases, conditions (a) - (d) are satisfied for the same set 
and injection. 

Case II: c = 2. By Lemma 16.61 we obtain that the tail of w is either 



X 



it n ^ fc+ina+1 .. n) (^ +1 (4 + i),^ + i(J fe ) U (0) if I + 1 6 M k+1 or 

= m k — 1. 

(d) are satisfied for the set M n ([I + 



^n,^ +1 (W<4U<m fc -l>) if I 

In the first case, conditions (a] 



l..n) x Z) and the injection ip' obtained by restricting p to this set. Note 
that M n ([/ + l..n) x Z) is a subset of 0^(4+1) = fi{^(^ +1 (4+l)) and 



Imp' is a subset of n (4+l) = SgjTjWft+l)) as required. 

In the second case, conditions (a) - (d) are satisfied for the same set and 
injection. 

Case III: c = 3 and Imp C ^m k ,n{{Ik+i)i+i^i) ■ Since lunp is a subset 
of _ 1 n ((4+i)z+iF->j) and 9? is strictly increasing, M is a subset of the inte- 



rior (see Section [5]) of E 



(d) 



mt-l,n 



((J fc+ i)j+w)-ThatisAfcn 



vnj,— 1,71 



((4+l) 



Z+li- 



Applying Lemma 16.71 for m' = m k , we obtain that the tail of w is ei- 

ther T mln,Mk+S( Ik +^ l + 1 » h Jfe) OT r iS^ fc+1 n( 9 ..n)(^ 9 ( 7 fe+l)Hl-/>^( J fc))> 

where q S (m k ..l + 1] fl A4+1- 
In the first case, conditions 



(d) are satisfied for the same set and 



injection. In the second case, conditions (a) - (d) are satisfied for the set 
M n ([q..n) x Z) and the injection p' obtained by restricting ip to this 

set. Note that M n {[q..n) x Z) is a subset of n ((4+i)z+ih->z) = 







(d) 

q— l,7i 



A:+lJ«+li-^i; 



and Im<// is a subset of £g d l((4+i)z+i 



T,( q d l(1Z q (I k+ i)i + i^i) as required. 



Case IF: c = 3 andlm<£ ^ £^, n ((4+i)z+i.-*.z)- Since E&^ )n ((4 



+1)1+1^1) 

E^, n (4+l) \ {(/ + 1, d- |/ fc+ i we have (l + l,d- |4+i | ( -°°"' ] ) G 

Imp. Therefore (l+l,d-\I k+1 ^-°°- 1 ^ = p((m',h')) for some point (m ', /V) € 
M. Consider the set M := M \ {(m',h')} and the injection p := <p\j^. 

We have Imv3 C E^, n ((4+i)z+:u-+i) and hence M C ^ mfc -i,n((4+i)m^z), 
since <p is strictly increasing. 
Applying property (c) , we get 

(C(m', Z+l)+u m ,-d+|I|(-°°-']) r = dist x ((m',h'),ip((m',h'))) = 
by property [(d)} Therefore, applying Lemma [6.71 (with origin at m'), we can 
assume that the tail of w is either %^, niM \{ m >y((Ik+i)l+l>->h J k ) if m k < 



m 



' or r in,A< fc+1 n( 9 ..n)(^(4+l)i+l^,^(^)), where 5 6 (m'.i + 1] C.M fc +l- 
In the first case, conditions (a) - (d) are satisfied for the set M and the 
injection tp. In the second case, conditions (a) - (d) are satisfied for the set 
Mfl ([</•■ ft ) x Z) and the injection obtained by restricting <p to this set. One 
can see this similarly to case III considering M instead of M and (p instead 
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of ip. Not that M n ([<?.. n) x Z) = M n ([g..n) x Z), whence condition [(a) 
follows. 

After considering all possible cases, we have prove that ev(w) T v^ £ V^. 



This implies property (ii) Also notice that V\ is homogeneous, that is the 



sum of its weight spaces. 
Lemma 7.1. V\ = 

Proof. Suppose that V\ is nonzero. Let v be a nonzero element of V\ of 
maximal weight. We can obviously assume that v = ev(u) T v^ for some 



T-monomial u satisfying conditions (a) - (d) 



Property (ii) implies then that v is a GL n _i(AT)-high weight vector. The 
irreducible module L n (X) can be realized as the socle of the module V n (A) 
contravariantly dual to the Weyl module with highest weight A. By [BKS, 
Corollary 3.3], we have v = c ■ for some \i € X + (n — 1) and c € K. 



Applying [BKS, Lemma 2.6(h)] and property (i) , we get 



cf x = £^ ) ...£t7 l) e^uYvt = 0, 

where u has weight — a\a\ ■ — a n -\a n -i. Hence c = and v = contrary 

to assumption. □ 
Remark. In this proof, we used the vectors and f\ introduced 
in |BKSl Section 2], where only the polynomial case (that is Ai • • • ^ A n 
0) was considered. However, one can pass to the general case, multiplying 
V n (A) by a power of the determinant representation. 



7.3. "If-part" of Theorem ll.il We are going to apply Lemma l5.4l to the 
X n i?o- Therefore, condition (ii) of Lemma HL4] is satisfied, whence there 



parameters a := i + 1, b := n, c := 0, d := d and X := Xj(i, n). Then under 
the notation of Lemma 15.41 we have Y := 2)^(i,n) and £ A (i,n) x {0} C 



exists a strictly decreasing injection ip : Y — ► X. We put M := Im^ and 
denote by p> the injection from M to Y inverting ip. 

We claim that 7^ m^) v X 1S a nonzero GL n _i(JC)-high weight vector of 
L n (A). Consider the T-monomial u = T^^^ M ^(0, Jq) and any homomor- 

phism r satisfying (|7.1|) . 

Take any I = i, . . . , n — 2. Clearly, (|7.6p holds in the present case. Lem- 
mas I6.5H6.7I imply that either pi (u) = or 

Pi(u)= P f\u) = -Tlfl h7ri(M)n(lA+1) {(l),J )x 

X/7 «+l,n,7ri(M)n(«+l..n)( ' 

where I + 1 € 7Ti(M). Note that pi(u) has weight ^ 0, since d > 0. The 



tail of p;(u) satisfies conditions (a) -(d) for the set M D ([I + l..n 



and the injection <p' obtained by restricting (p to this set. Note that M C 
(i..n) x [0..d), whence M n ([/ + l..n) x Z) C (Z..n) x [0..d) = 

Moreover, Im<^' C (Z+L.n] X (0..d] C sS n (0) and dist>(;c, y) = (mod p) 
for any x,y G A. Thus applying (|7.6|) . we obtain 

^ ^Jm(0)4 = ( eV e V A- 
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By Lemma 17.11 we have £/ ^jf ^(0)^ = 0- Since d < p, we have 
£ l T),J i%M ( ) v x = for any r > and / = 1, . . . , n - 2. 



In the present case, (|7,2p holds and takes the form 

d! • • • • ^^.M^K = cf = cf.(n)^+ (7.7) 

where x = (d,0 n ~ i_1 ) by virtue of (fTTH) . where J fc+ i = and J fe = ((i-l) d ). 
We put t(t) := (p((t,h)) for any (t, /i) G M and apply Lemma fo.191 for this 

injection. Note that E^j (0) = (i.n]x[0..d] and A£ n (J ) = ({i+l}x [0..d]) U 

((i+l..n]x{0}). Therefore, Im t = F C (»+l..n]x(0..d] = X^(0)\A^ n (J o ) 
as the hypothesis of Lemma 16.191 requires. Again, we have ()7.5j) . which in 
the present case takes the form 

tf„(u) T v+ = d\(Xi - A m )^x 

x n{^~ * + K-h-h\ (t,h) e(i + l..n] x (0..d\ \lmi}v% 



= d\ n{dist A ((i, 0), x) I x G (i..n\ x (0..d] \ Y}v+ + 

by the definition of Y. Hence "ji% M (0)v% / by (jUj) . 

7.4. Some basis coefficients. Let / be a multiset of length no more than 
d with entries in [i..n). We put 

p(d) ._ (Y[ n ^ p^\ {t} )\ p( rf -l 7 D 
*i, n ,i — \LL t =i+i ^ / 4 >™ ' 

This element has weight A^ (I, Jo). 

Lemma 7.2. Suppose M C f2^(J). T/ien modulo the ideal X ofU° gen- 
erated by the polynomials ut — h, where (t, h) G M, the F^ ^coefficient of 
ev (^n?7ri(M)(^' J o)) is equivalent to 

di (rCn i j i m! ) n{^) - * 1 e \ m }- 



Proof. Induction on |M|. For M = 0, the result follows from (13.50 . 

Now suppose M 7^ 0. Let x = (m, fj) be the element of M with smallest 
first coordinate. By the inductive hypothesis, modulo X, the Xy^ ^-coefficient 

of ev (^^ 1 (M\{ a; })( / ' J o)) is equivalent to 

dl {UZL |/|W! ) U{ C ^ ~ h I G \ ( M \ M)}- (7-8) 

Since X is stable under ai >m , it follows from this equivalence that the F^ j- 
coefficient of ^i.m^ev^^^^^I, J ))) is equivalent modulo X to 

* ac (it:;, icr*-* - ») 

x H{c(m, *) - h I (t, fc) G n£g n (I) \ (M \ {x})}, 

which belongs to X. More exactly, u m d ~^ G X, since 

< f> < d- |J|C-«>»n»]. Hence by Lemma EJ3 the ^^-coefficient of 
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ev(7~}* Jo)) (C(i, m) — u m ) is equivalent modulo X to (|7.8|) . which in 
turn is equivalent to 

dl {UZL U{ C ^ - h I (*» h ) G n S( J ) \ M}(C(i,m) - u m ). 

From Proposition l6.1^n)| it follows that C(i, m)—u m £ I. Now the required 
result follows from Proposition 16. lq[ ill □ 



Proposition 7.3. Let N € UT(n) and 1 ^ j < n. Then we have 

F j:n F {N ^ = (N j:n + \)F^ N+e ^ + J^~*(iVk,„ + l)F^- e w +e *.»). 

Let Vi denote the Z'-submodule of U~ spanned by all F^ N \ where N 
belongs to UT^°(n) and has in all rows except row i and W< denote the 
Z'-submodule of U~ spanned by all other F^ N > . We have U~ = Vi ffi W;. 
We denote by proj the projection of U~ to the first summand. 

Remark. Proposition 17.31 implies that Wi is closed under right multipli- 
cation by Fj :U . 

Lemma 7.4. Let 1 ^ j < n, I be a multiset with entries in [i..n) of length 
no greater than d and j occur in I . Then we have 

P^m,nF^) = (d-\L\ + l)F^ AU) . 

Proof. Let N be the matrix of UT>°(n) such that F$j = F^ N \ Since N 
has in all rows distinct from row i, we can assume that the summation 
parameter k in Proposition 17.31 takes the only value %. 

Consider the case j = i. Then only the first summand remains, whence 
proj(F 7> F( d )) = (N ijn + l)F( N+ei < n \ Now consider the case j > i. Then 
only the second summand contributes to Vi and we obtain pio][Fj n F^) = 
(N i>n + l)F( N ~ e ^+ e ^. □ 

For a multiset / with entries in [i..n) of length no greater than d, we define 

c-(n:: 1 < /|< " ) 



t=i 

(d) 



which may be viewed as a complimentary element for F i ' 



i 



Corollary 7.5. Let p be a prime greater than d and I be a multiset with 
entries in [i..n) of length no greater than d. Then the F^ -coefficient of 

^in I^in I * s an * n ^ e ff er n °t divisible by p. 

Lemma 7.6. iy?r is the only element F^ , where N G UT^°(n), of given 
weight belonging to Vi. 

Proof. The sums of elements in columns i + 1, . . . , n can be expressed via 
the weight of iy?r by (|2,ip and (|3.3|) . However, each of this columns has 
everywhere but in row i. □ 
We denote by 9^„j and the images of F^i and F> d ^j in L^(ra) 

respectively. 
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7.5. "Only if-part" of Theorem 11.11 Without loss of generality we may 
suppose that Ai ^ • • • ^ A n ^ 0. We need this stipulation to apply the 
theory developed in [BKSJ . 

We apply induction on n — i. Suppose first that n — i = 1. Then 



Since d < p, it follows from the above formula that A ra _i — A ra — q ^ 
(mod p) for any q = 0, . . . , d — 1. Hence 2)^(«, n) = 0, as required. 

Now suppose that n — i > 1. Then there exists some F G U^(n) of weight 
—da(i, n) such that is the a nonzero GL n _i(i ; C)-high weight vector. Re- 
call that L n (A) can be realized as the socle of the module V n (A). Therefore 
E (d) _ _ gW 1 i? u + ^ o by [BKSl Lemma 2.6(h)]. In particular, ^ °- 

Using the commutator formulas for Uxin), we obtain E^F = F'H + E, 
where F' is an element of U^(n — 1) having weight — da(i,n— 1), if G C/^(n) 
and E belongs to the left ideal of f/^(n) generated by the elements for 
r = 1, . . . , d. Hence F'Hv^ = E,!^_ 1 Fvt ^ 0. We have Hv~^ = c-v^ for some 
c £ K. Therefore F'v^ ^ and c / 0. Finally, take any s = 1, . . . , n — 3 
and r > 0. We have 



We have proved that F'vt is a nonzero GL n _2(-ftT)-high weight vector 
of weight A — da(i,n — 1) belonging to the module U^(n — which is 

isomorphic to L„_i(Ai, . . . , A„,_i) by [Sin] (this can be seen directly). 




Applying the inductive hypothesis, we obtain that for each subset A of 
2)^(i,n — 1) whose points are incomparable with respect to <, there ex- 
ists a strictly decreasing injection from A to <£ A (i,n — 1) x {0}. Apply- 
ing Lemma 15.41 similarly to how we did it in Section 17.31 we obtain that 
there exists a strictly decreasing injection ip : 2)^(i, n — 1) — > X£(i,n — 1). 
We endow the set X$(i,n — 1) with the nonstrict partial order =^ as in 
Lemma 15.41 That is x ^ y if and only if there are elements zq, . . . , z m , 
where m ^ 0, of X£(i, n — 1) such that y = zq, x = z m and z s = ip(z s -i) for 
any s = 1, . . . , m. Following the proof of Lemma 15.41 we put ip{y) to be the 
smallest (w.r.t. =4) element of {x G X \ x =^ y}. Thus ?/> maps X^(i,n — 1) 
to X£(i,n — 1) n ((z..n — 1] x {0}), which equals £ x (i,n) x {0}. Moreover, 
ij)[x) = ifj(x') if and only if x and x' are comparable with respect to 

Suppose that there exists a subset T of 2)^(i,n) whose points are incom- 
parable with respect to < such that there is no strictly decreasing injection 
from r to £ x (i,n) x {0}. By Lemma l5.7l there exists a multiset / of length 
no greater than d with entries in the interval [i..n) such that T is contained 
in the boundary of S^(J). Note that F contains a point z' in column n, 
since otherwise T C %)j(i,n — 1). 



Consider the set M := U^(I) n X£(i,n- 1). Take a point x G M. We 



CxX 0. Hence 





claim that there exists a point y G £• ^(1) n 



(i, n — 1) such that x = V'(y)- 
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Indeed, consider the map £ : T — > £ x (i,n) x {0} denned by 

\ ip(x) if z = z', 

where zGT, Observing that i/j(x)<z' , we obtain that £ is strictly decreasing. 
Hence £ is no injection. This is only possible if vp(x) = ip{x') for some i'gT 
not belonging to column n. In that case, the points x and x' are comparable 
with respect to Note that x' is a boundary point of and x is an 

interior point of this set. Therefore by definition there are points zq, . . . ,z m 
of Xj(i,n — 1) such that m > 0, x' = zq, x = z m and z s = ijj(z s _i) for any 

s = 1, . . . , m. It is easy to notice that Zo,...,z m belong to Thus 
we can take y := z m -\. 

Let ip : {(j,0)}UM -> £^(J) nA$(i,n) be the map such that <p{(i,0)) = 
z' and (p(x) = y if x £ M and x = ip{y). Obviously, ip is a strictly 
increasing injection. Also note that {(«', 0)} U M C £;1\ and thus 
{(i, 0)}UM C 0^ 1>n (I), since 99 is strictly increasing. Hence 1{^, M ^(I, Jo) 
satisfies conditions (a) - (d) of Section \77I\ for the set {(i,0)} U M and the 
injection ip, where r defined by (I7.ip . Therefore m(I) v \ e ^A- Applying 
Lemma EU we obtain "Jj® m(^) v X = 0- 

Lemma l?T2l implies that the 3^ ^-coefficient of X^? jy-(-f) * s 

* (eCw ^) n{* - f + ^ - ^ - * 1 (*» h ) e "S w \ m }- 

Applying Corollary 17. 51 and Lemma [7. 61 along with the remark of Section [7.4l 
we obtain that 3^ -coefficient of ^fni^inM^) ls 

c-]j{i-i + Wi-0<t-h\(t,h)enl d l(I)\M}, (7.9) 

where c is a nonzero element of K. 

Denote by S the element of Uj^{n) obtained from 9^ j-li ^ written 
in the basis of Section 12.51 by the specialization Jit l— * \ f° r * = 1, • • • , 
Our choice of M and (|7.9|) ensure that the 3"- ^-coefficient of <S is nonzero. 
However, 

svi = $%,ri% M m = 0. 

Applying |BKSl Lemma 3.6] for \i = (Ai, . . . , Aj_i, Aj - d, A i+ i, . . . , A n _i), 
we obtain that there is no nonzero GL n _i(i^)-high weight vector of weight 
fx in L n {\) (/i is not normal for A). This is a contradiction. 

Remark. The condition d < p is necessary in Theorem ll.il Indeed, if we 
take, for example, d = p, i = n — 1 and A € X + (n) such that A„_i — X n = 
2p — 1, then we obtain that the vector 3^^ is a nonzero GL n _i (i^ )-high 
weight vector, since 

c( d ) rrW „+ _ / An-l-AnV + _ / 2p-l\ + _ + 
C n-l J n-l,n V \ ~\ d ) V \ ~\p ) A ~~ U A ' 

However, <£ A (i,n) = and (n,p) € 2)^(i,n). 
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8. Appendix: List of Notations 
K algebraically closed field of characteristic p > 0; 

X + (n) {A € Z n : Ai ^ • • • ^ A n }, set of dominant weights; 

L n {\) irreducible GL n {K )-module with highest weight A; 

fixed nonzero vector of L n (\) of weight A; 
[s..t], [s..t), etc. {x E Z | s ^ x ^ t}, {x G Z | s ^ x < t}, etc., p. [Q 
(a, 6) < (x,y) a < x and b < y; 

?)d0> n) {(*, M G (l.n] x \ t - i + Xi - X t - h = (mod p)}; 

(£ A (z, n) {s € (i.n) |s — z + Aj — A s = (mod p)}; 

a(s, t) (0, . . . , 0, 1, 0, . . . , 0, —1, . . . , 0) of length n with 1 at posi- 

tion s and —1 at position t; 

7Ti map from Z 2 to Z taking (s,t) to s; 

ao (— 1, 0, . . . , 0) of length n; 

a t a(t,t + l); 

x— x- ■ ■ (x — n + 1) if n ^ and l/(x + 1) • • • (x — n) if n < 0; 

a m sequence of length m whose every entry is a; 

AuB = C AllB = C and Af]B = 0; 
\S\ cardinality of set S; 

5-p 1 or if V is true or false respectively; 

UT(n) set of integer nx n matrices N such that N a ^ = unless 

a < b; 

UT^°(n) subset of UT(n) consisting of matrices with nonnegative 

entries; 

eij,Xij nx n matrix with 1 at the ij'th position and elsewhere; 

Nt Sa=i N a> t sum of elements in column t of N, p. [U 

N s J2b=i N s ,b sum of elements in row s of N, p. HJ 

N a , b ,p.® 

dist A (x,y) yi-x 1 + \ Xl -\ yi +x 2 -y2 if x = (xi,x 2 ) and y = (yi,y 2 ), 

X^(i,n) {x G (i..ra] x [0..J] | dist A ((i, 0), x) =0 (mod p)}; 

\I\ S number of elements of multiset / belonging to set S; 

\I\ total number of elements of multiset /; 

multiset obtained from / by replacing one x with y; 

C m (J) (min{ji,m - 1}, . . . ,min{j fc ,m - 1}} if J = . . ,j k ); 

R-m{J) (js | s = 1, . . . , k, j s ^ m - 1} if J = 0i, ...,jk); 

C m (I) (i s \s = l,...,l, i s <m-l) if /=(!!,...,»,); 

7Z m (I) (maxjii, m — 1}, ... , max{i/, m — 1}) if / = (ij, . . . , ii); 

Ui 0; 

Q' Q(uj + i, . . . , n n _i) field of rational fractions, p. 

-^s AC SS , 

il(g/q/(n)) universal enveloping algebra of QlQ>(n), p. [5j 
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u 

TJ 

( x r) 

u 

^s,! > r s,t 

eP 
f( n ) 

w 

I + ,I + 

u° 

u~ 

u-v 
u->° 

C(k,l) 
K T 

U K (n) 

T 

x 

pW rr(r) 

U° K (n) 
U K (n) 



A 



id) 

,n,M 



°i,n,M 
p(d) 
i,n,M 

cone(x) 

cone(S') 

(a, b) < (x,y) 

snake(T) 



subring of il(g/q/(re)) generated by H\, . . . ,H n , 
Ui+l, ■ ■ -,Un-l, P-El 

right (left) ring of quotients of il(g/q/(n)) with respect to 

u°\{o},p.m 

Z[iij + i, . . . , u n _i] polynomial algebra over Z; 
(X St t) r /r\, where s ^ t, 1 ^ s, i ^ re and r ^ 0, p. EJ 

• • • (X S]S — r + l)/r!, where 1 ^ s ^ n and r ^ 0, p. 
Z'-subalgebra of lt(g/Q/(n)) generated by X^l and ( X ^' s ); 
X^t ,Xt s respectively, where 1 ^ s < t ^ re; 



(r) 



sub field of U generated by U°; 
left ideals of U and C/ resp. generated by E s 
Z'-subalgebra of U generated by all 



(r) 



with r > 0; 



Z -subalgebra of U generated by all F s / ; 

(r) 

Z -subalgebras of U generated by all F s / and H s ; 
subring of U generated by U~ and U°; 
l-k + H k -Hf, 

field K considered as a left Z'-module via the multiplica- 
tion rule f • a = r(f)a, where / £ Z' and a G if, p.0 

C7 <8>z' hyperalgebra over filed K, p. [2 

x (g> Ik, image of x G [7 in UK(n) = U <S>z' K T under r, 

P- El 

images of and respectively in Uk{ti), p. El 
images of F^ N > and if s respectively in Uk(ji), p. El 

if -subalgebra of Uk(ti) generated by all 
if -subalgebra of Uk (re) generated by all ; 
eq. (HH), p. El 
eq. $EM, p.EB 
eq. flU]) and (JO}, p. 15]- 16; 
set of elements less than or equal to x, pp. [22j [24l 

a ^ x and 6 ^ y; 
boundary of cone(r), p. [2U 

{{t,h) eZ 2 jkt^'&oa^- |j|(-oo..t-i]| 5 p 
{(t,h) eZ 2 \k<t<jk0^h<d - |/|(-°°"*]}, p 
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0"Z,m P- __3 

Sm m r(I>J) formal operator, p. 

algebra of formal operators, p. 

pf ] p. EBB 

(2,1,) (2,R) (3) pjYi 

Pi , Pi , Pi ; P - ED 

(2) (2,L) (2,R) 

T k d j,M ( 7 > J ) Definition El p. EH 
cC eq. i3D, p.HD 

cfg-^J) p.H2 

ev evaluation function, p. HH 

cf(-P) sum of cf >< (P) over all sequences of nonnegative integers 

x of length n — i, p. 05J 

(1) , (2) , (3) 
PI Pl'+Pl'+Pl'] 

A k,j( J ) {{t,h)eZ 2 \k<t^j&i\4 t -- +0O Hh^\J\^- 1 -- +oo ^+q t }, p.Ha 

fh Ik + • • • + Ik with m summands; 

proj projection of ?7 to Vj along Wj, p. ESI 

(llS^V-S 

^i,n,/' ^i,n,7 images of and in [/^(n) respectively, p. EH 
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